2025-12-04T09:19:37.1790063Z Current runner version: '2.330.0' 2025-12-04T09:19:37.1795243Z Runner name: 'i-0cbeb234d1c75d5ac' 2025-12-04T09:19:37.1795867Z Runner group name: 'default' 2025-12-04T09:19:37.1796670Z Machine name: 'ip-10-0-31-90' 2025-12-04T09:19:37.1798710Z ##[group]GITHUB_TOKEN Permissions 2025-12-04T09:19:37.1801009Z Contents: read 2025-12-04T09:19:37.1801447Z Metadata: read 2025-12-04T09:19:37.1801828Z ##[endgroup] 2025-12-04T09:19:37.1803577Z Secret source: Actions 2025-12-04T09:19:37.1804176Z Prepare workflow directory 2025-12-04T09:19:37.2211587Z Prepare all required actions 2025-12-04T09:19:37.2244875Z Getting action download info 2025-12-04T09:19:37.5820304Z Download action repository 'pytorch/test-infra@main' (SHA:39aa74d619174326f4e2fb0e216151c2f29d9ffd) 2025-12-04T09:19:40.3046173Z Download action repository 'pytorch/pytorch@main' (SHA:7716da9fb23f27a65b41f9f016a2afadf281c18f) 2025-12-04T09:19:57.2879772Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-12-04T09:19:57.6692906Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-12-04T09:19:57.9410668Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-12-04T09:19:58.1621299Z Download action repository 'seemethere/download-artifact-s3@1da556a7aa0a088e3153970611f6c432d58e80e6' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T09:19:58.3991312Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T09:19:58.6959891Z Getting action download info 2025-12-04T09:19:58.8359447Z Download action repository 'actions/checkout@v4' (SHA:34e114876b0b11c390a56381ad16ebd13914f8d5) 2025-12-04T09:19:59.1899654Z Getting action download info 2025-12-04T09:19:59.3214930Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-12-04T09:19:59.5363131Z Getting action download info 2025-12-04T09:19:59.6729462Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-12-04T09:19:59.8673345Z Getting action download info 2025-12-04T09:20:00.0247780Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32) 2025-12-04T09:20:00.0251099Z ##[group] Inputs 2025-12-04T09:20:00.0251713Z build-environment: linux-jammy-py3.10-gcc11-build 2025-12-04T09:20:00.0258564Z test-matrix: {"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]} 2025-12-04T09:20:00.0265306Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:20:00.0265949Z sync-tag: 2025-12-04T09:20:00.0266819Z timeout-minutes: 240 2025-12-04T09:20:00.0267037Z use-gha: 2025-12-04T09:20:00.0267279Z dashboard-tag: 2025-12-04T09:20:00.0267538Z s3-bucket: gha-artifacts 2025-12-04T09:20:00.0267797Z aws-role-to-assume: 2025-12-04T09:20:00.0268249Z disable-monitor: false 2025-12-04T09:20:00.0268536Z monitor-log-interval: 5 2025-12-04T09:20:00.0268816Z monitor-data-collect-interval: 1 2025-12-04T09:20:00.0269118Z ##[endgroup] 2025-12-04T09:20:00.0269565Z Complete job name: periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:20:00.0736634Z A job started hook has been configured by the self-hosted runner administrator 2025-12-04T09:20:00.0814763Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-12-04T09:20:00.0822228Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:20:00.0822656Z ##[endgroup] 2025-12-04T09:20:01.0779856Z Runner Type: linux.8xlarge.amx 2025-12-04T09:20:01.0780340Z Instance Type: m7i-flex.8xlarge 2025-12-04T09:20:01.0780539Z AMI Name: unknown 2025-12-04T09:20:01.0884415Z AMI ID: ami-08982f1c5bf93d976 2025-12-04T09:20:05.4648686Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-12-04T09:20:05.4648986Z with: 2025-12-04T09:20:05.4649505Z github-secret: *** 2025-12-04T09:20:05.4649942Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-12-04T09:20:05.4650391Z activate-with-label: false 2025-12-04T09:20:05.4650578Z label: with-ssh 2025-12-04T09:20:05.4650740Z remove-existing-keys: true 2025-12-04T09:20:05.4651131Z fail-silently: true 2025-12-04T09:20:05.4651317Z env: 2025-12-04T09:20:05.4651629Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:20:05.4651844Z ##[endgroup] 2025-12-04T09:20:05.5896536Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-12-04T09:20:05.5902013Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-12-04T09:20:05.6025034Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-12-04T09:20:05.6025300Z with: 2025-12-04T09:20:05.6025455Z no-sudo: true 2025-12-04T09:20:05.6025616Z submodules: recursive 2025-12-04T09:20:05.6025779Z fetch-depth: 0 2025-12-04T09:20:05.6025926Z env: 2025-12-04T09:20:05.6026070Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:20:05.6026233Z ##[endgroup] 2025-12-04T09:20:05.6084602Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:20:05.6085186Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:20:05.6093091Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:20:05.6093330Z env: 2025-12-04T09:20:05.6093524Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:20:05.6093733Z ##[endgroup] 2025-12-04T09:20:05.6164482Z ##[group]Run # Use all available CPUs for fetching 2025-12-04T09:20:05.6164794Z # Use all available CPUs for fetching 2025-12-04T09:20:05.6165017Z cd "${GITHUB_WORKSPACE}" 2025-12-04T09:20:05.6165247Z git config --global fetch.parallel 0 2025-12-04T09:20:05.6165500Z git config --global submodule.fetchJobs 0 2025-12-04T09:20:05.6165725Z  2025-12-04T09:20:05.6166035Z # Clean workspace. The default checkout action should also do this, but 2025-12-04T09:20:05.6166364Z # do it here as well just in case 2025-12-04T09:20:05.6166577Z if [[ -d .git ]]; then 2025-12-04T09:20:05.6166788Z  if [ -z "${NO_SUDO}" ]; then 2025-12-04T09:20:05.6166987Z  sudo git clean -ffdx 2025-12-04T09:20:05.6167175Z  else 2025-12-04T09:20:05.6167342Z  git clean -ffdx 2025-12-04T09:20:05.6167512Z  fi 2025-12-04T09:20:05.6167659Z fi 2025-12-04T09:20:05.6172010Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:20:05.6172529Z env: 2025-12-04T09:20:05.6172733Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:20:05.6172914Z NO_SUDO: true 2025-12-04T09:20:05.6173064Z ##[endgroup] 2025-12-04T09:20:05.6270580Z ##[group]Run actions/checkout@v4 2025-12-04T09:20:05.6270814Z with: 2025-12-04T09:20:05.6271007Z ref: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:20:05.6271252Z fetch-depth: 0 2025-12-04T09:20:05.6271428Z submodules: recursive 2025-12-04T09:20:05.6271609Z show-progress: false 2025-12-04T09:20:05.6271802Z repository: pytorch/pytorch 2025-12-04T09:20:05.6272094Z token: *** 2025-12-04T09:20:05.6272257Z ssh-strict: true 2025-12-04T09:20:05.6272770Z ssh-user: git 2025-12-04T09:20:05.6272958Z persist-credentials: true 2025-12-04T09:20:05.6273151Z clean: true 2025-12-04T09:20:05.6273326Z sparse-checkout-cone-mode: true 2025-12-04T09:20:05.6273536Z fetch-tags: false 2025-12-04T09:20:05.6273703Z lfs: false 2025-12-04T09:20:05.6273864Z set-safe-directory: true 2025-12-04T09:20:05.6274058Z env: 2025-12-04T09:20:05.6274218Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:20:05.6274393Z ##[endgroup] 2025-12-04T09:20:05.7202382Z Syncing repository: pytorch/pytorch 2025-12-04T09:20:05.7203399Z ##[group]Getting Git version info 2025-12-04T09:20:05.7203708Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T09:20:05.7204122Z [command]/usr/bin/git version 2025-12-04T09:20:05.7409029Z git version 2.50.1 2025-12-04T09:20:05.7425607Z ##[endgroup] 2025-12-04T09:20:05.7442904Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/0b5891cb-7e4d-46b0-bd16-c4651c85eb03/.gitconfig' 2025-12-04T09:20:05.7467148Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/0b5891cb-7e4d-46b0-bd16-c4651c85eb03' before making global git config changes 2025-12-04T09:20:05.7467965Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T09:20:05.7471407Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:20:05.7528206Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T09:20:05.7529189Z ##[group]Initializing the repository 2025-12-04T09:20:05.7536390Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:20:05.7608300Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-12-04T09:20:05.7609024Z hint: is subject to change. To configure the initial branch name to use in all 2025-12-04T09:20:05.7609421Z hint: of your new repositories, which will suppress this warning, call: 2025-12-04T09:20:05.7609705Z hint: 2025-12-04T09:20:05.7609937Z hint: git config --global init.defaultBranch 2025-12-04T09:20:05.7610163Z hint: 2025-12-04T09:20:05.7610386Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-12-04T09:20:05.7610745Z hint: 'development'. The just-created branch can be renamed via this command: 2025-12-04T09:20:05.7611019Z hint: 2025-12-04T09:20:05.7611173Z hint: git branch -m 2025-12-04T09:20:05.7611354Z hint: 2025-12-04T09:20:05.7611838Z hint: Disable this message with "git config set advice.defaultBranchName false" 2025-12-04T09:20:05.7617221Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-12-04T09:20:05.7626975Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-12-04T09:20:05.7662373Z ##[endgroup] 2025-12-04T09:20:05.7662756Z ##[group]Disabling automatic garbage collection 2025-12-04T09:20:05.7666178Z [command]/usr/bin/git config --local gc.auto 0 2025-12-04T09:20:05.7692129Z ##[endgroup] 2025-12-04T09:20:05.7692691Z ##[group]Setting up auth 2025-12-04T09:20:05.7697189Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T09:20:05.7726077Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T09:20:05.8058669Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T09:20:05.8087592Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T09:20:05.8404149Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:20:05.8433573Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T09:20:05.8733122Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T09:20:05.8803881Z ##[endgroup] 2025-12-04T09:20:05.8806811Z ##[group]Fetching the repository 2025-12-04T09:20:05.8810344Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-12-04T09:20:54.2597510Z From https://github.com/pytorch/pytorch 2025-12-04T09:20:54.2600438Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-12-04T09:20:54.2601154Z * [new branch] 2.9.1 -> origin/2.9.1 2025-12-04T09:20:54.2601523Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-12-04T09:20:54.2601911Z * [new branch] Flamefire-patch-1 -> origin/Flamefire-patch-1 2025-12-04T09:20:54.2602594Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-12-04T09:20:54.2602985Z * [new branch] HOPrintFunc -> origin/HOPrintFunc 2025-12-04T09:20:54.2603312Z * [new branch] IvanKobzarev/stack/1 -> origin/IvanKobzarev/stack/1 2025-12-04T09:20:54.2603823Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-12-04T09:20:54.2604162Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-12-04T09:20:54.2604541Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-12-04T09:20:54.2604897Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-12-04T09:20:54.2605239Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-12-04T09:20:54.2605558Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-12-04T09:20:54.2605923Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-12-04T09:20:54.2606248Z * [new branch] VLA_exp -> origin/VLA_exp 2025-12-04T09:20:54.2606550Z * [new branch] activation_bench -> origin/activation_bench 2025-12-04T09:20:54.2606859Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-12-04T09:20:54.2607175Z * [new branch] adi/onednn_aarch64 -> origin/adi/onednn_aarch64 2025-12-04T09:20:54.2607478Z * [new branch] adi/test -> origin/adi/test 2025-12-04T09:20:54.2607768Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-12-04T09:20:54.2608064Z * [new branch] adi/test_m8g -> origin/adi/test_m8g 2025-12-04T09:20:54.2608368Z * [new branch] adi/test_onednn -> origin/adi/test_onednn 2025-12-04T09:20:54.2608705Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-12-04T09:20:54.2609064Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-12-04T09:20:54.2611860Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-12-04T09:20:54.2615796Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-12-04T09:20:54.2616258Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-12-04T09:20:54.2616642Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-12-04T09:20:54.2617142Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2025-12-04T09:20:54.2617509Z * [new branch] also-surround-shimh -> origin/also-surround-shimh 2025-12-04T09:20:54.2617877Z * [new branch] angelayi/aot_compile -> origin/angelayi/aot_compile 2025-12-04T09:20:54.2618278Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-12-04T09:20:54.2618672Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-12-04T09:20:54.2619107Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-12-04T09:20:54.2619529Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-12-04T09:20:54.2619893Z * [new branch] angelayi/inductor_const -> origin/angelayi/inductor_const 2025-12-04T09:20:54.2620244Z * [new branch] angelayi/lstm -> origin/angelayi/lstm 2025-12-04T09:20:54.2620656Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-12-04T09:20:54.2621533Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-12-04T09:20:54.2622423Z * [new branch] angelayi/side_eff -> origin/angelayi/side_eff 2025-12-04T09:20:54.2622919Z * [new branch] angelayi/state_dict -> origin/angelayi/state_dict 2025-12-04T09:20:54.2623568Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-12-04T09:20:54.2626390Z * [new branch] angelayi/symm_mem -> origin/angelayi/symm_mem 2025-12-04T09:20:54.2626825Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-12-04T09:20:54.2627175Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-12-04T09:20:54.2627553Z * [new branch] annotate_assert -> origin/annotate_assert 2025-12-04T09:20:54.2628133Z * [new branch] annotate_fallback_kernel -> origin/annotate_fallback_kernel 2025-12-04T09:20:54.2629090Z * [new branch] annotation_deepcopy -> origin/annotation_deepcopy 2025-12-04T09:20:54.2629512Z * [new branch] annotation_dynamo -> origin/annotation_dynamo 2025-12-04T09:20:54.2629862Z * [new branch] aot_eager_stack_trace -> origin/aot_eager_stack_trace 2025-12-04T09:20:54.2630389Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-12-04T09:20:54.2630774Z * [new branch] aoti_const_device -> origin/aoti_const_device 2025-12-04T09:20:54.2636208Z * [new branch] aoti_fqn_name_interface -> origin/aoti_fqn_name_interface 2025-12-04T09:20:54.2636649Z * [new branch] aoti_package_weights_binary -> origin/aoti_package_weights_binary 2025-12-04T09:20:54.2637006Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-12-04T09:20:54.2637393Z * [new branch] arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling 2025-12-04T09:20:54.2637757Z * [new branch] async_tp -> origin/async_tp 2025-12-04T09:20:54.2638126Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-12-04T09:20:54.2638547Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-12-04T09:20:54.2638935Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-12-04T09:20:54.2639256Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-12-04T09:20:54.2639570Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-12-04T09:20:54.2639923Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-12-04T09:20:54.2640220Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-12-04T09:20:54.2640803Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-12-04T09:20:54.2641200Z * [new branch] atalman-patch-8 -> origin/atalman-patch-8 2025-12-04T09:20:54.2642854Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-12-04T09:20:54.2643511Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-12-04T09:20:54.2643894Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-12-04T09:20:54.2644319Z * [new branch] attention_benchmarking_clean -> origin/attention_benchmarking_clean 2025-12-04T09:20:54.2645948Z * [new branch] bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add 2025-12-04T09:20:54.2646361Z * [new branch] bahuang/fix_debug_mode -> origin/bahuang/fix_debug_mode 2025-12-04T09:20:54.2646720Z * [new branch] bahuang/fix_expand -> origin/bahuang/fix_expand 2025-12-04T09:20:54.2647300Z * [new branch] bahuang/test -> origin/bahuang/test 2025-12-04T09:20:54.2648625Z * [new branch] base/1.5 -> origin/base/1.5 2025-12-04T09:20:54.2649265Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-12-04T09:20:54.2649959Z * [new branch] bench_scaled_mm_ops -> origin/bench_scaled_mm_ops 2025-12-04T09:20:54.2650711Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-12-04T09:20:54.2651351Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-12-04T09:20:54.2653148Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-12-04T09:20:54.2653636Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-12-04T09:20:54.2655261Z * [new branch] bf/bug-static-input -> origin/bf/bug-static-input 2025-12-04T09:20:54.2655613Z * [new branch] bf/cg-backend -> origin/bf/cg-backend 2025-12-04T09:20:54.2656001Z * [new branch] bf/cg-nccl-test -> origin/bf/cg-nccl-test 2025-12-04T09:20:54.2656650Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-12-04T09:20:54.2657403Z * [new branch] bf/clean-torchbench-hf -> origin/bf/clean-torchbench-hf 2025-12-04T09:20:54.2658053Z * [new branch] bf/combo-debug-log -> origin/bf/combo-debug-log 2025-12-04T09:20:54.2658707Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-12-04T09:20:54.2663483Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-12-04T09:20:54.2664178Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-12-04T09:20:54.2664726Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-12-04T09:20:54.2665115Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-12-04T09:20:54.2665514Z * [new branch] bf/dynamo-partition -> origin/bf/dynamo-partition 2025-12-04T09:20:54.2665829Z * [new branch] bf/lite -> origin/bf/lite 2025-12-04T09:20:54.2666150Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-12-04T09:20:54.2666538Z * [new branch] bf/partition-cache-free-symbols -> origin/bf/partition-cache-free-symbols 2025-12-04T09:20:54.2666956Z * [new branch] bf/partition-memory-plan -> origin/bf/partition-memory-plan 2025-12-04T09:20:54.2667325Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-12-04T09:20:54.2667682Z * [new branch] bf/partition-view-fallback -> origin/bf/partition-view-fallback 2025-12-04T09:20:54.2668053Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-12-04T09:20:54.2668658Z * [new branch] bf/timm-nov-26-2025 -> origin/bf/timm-nov-26-2025 2025-12-04T09:20:54.2669133Z * [new branch] bf/transformer-pin-4-57-3 -> origin/bf/transformer-pin-4-57-3 2025-12-04T09:20:54.2669780Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-12-04T09:20:54.2670427Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-12-04T09:20:54.2671053Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-12-04T09:20:54.2671690Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-12-04T09:20:54.2672581Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-12-04T09:20:54.2673410Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-12-04T09:20:54.2674062Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-12-04T09:20:54.2674633Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-12-04T09:20:54.2675328Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-12-04T09:20:54.2677932Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-12-04T09:20:54.2678429Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-12-04T09:20:54.2678789Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-12-04T09:20:54.2679164Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-12-04T09:20:54.2679774Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-12-04T09:20:54.2680857Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-12-04T09:20:54.2681309Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-12-04T09:20:54.2681751Z * [new branch] brister/fx_device_type -> origin/brister/fx_device_type 2025-12-04T09:20:54.2682369Z * [new branch] brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx 2025-12-04T09:20:54.2683007Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-12-04T09:20:54.2683604Z * [new branch] bwd-backup -> origin/bwd-backup 2025-12-04T09:20:54.2684049Z * [new branch] c57382a49 -> origin/c57382a49 2025-12-04T09:20:54.2685082Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-12-04T09:20:54.2685935Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-12-04T09:20:54.2686679Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-12-04T09:20:54.2687456Z * [new branch] cccclai-patch-1 -> origin/cccclai-patch-1 2025-12-04T09:20:54.2688191Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2689008Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2689848Z * [new branch] cherry-pick-162208-by-pytorch_bot_bot_ -> origin/cherry-pick-162208-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2690571Z * [new branch] cherry-pick-163169-by-pytorch_bot_bot_ -> origin/cherry-pick-163169-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2691355Z * [new branch] cherry-pick-165086-by-pytorch_bot_bot_ -> origin/cherry-pick-165086-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2692281Z * [new branch] cherry-pick-165514-by-pytorch_bot_bot_ -> origin/cherry-pick-165514-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2692998Z * [new branch] cherry-pick-165601-by-pytorch_bot_bot_ -> origin/cherry-pick-165601-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2695254Z * [new branch] cherry-pick-165667-by-pytorch_bot_bot_ -> origin/cherry-pick-165667-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2695992Z * [new branch] cherry-pick-165815-by-pytorch_bot_bot_ -> origin/cherry-pick-165815-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2696625Z * [new branch] cherry-pick-165922-by-pytorch_bot_bot_ -> origin/cherry-pick-165922-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2697116Z * [new branch] cherry-pick-166148-by-pytorch_bot_bot_ -> origin/cherry-pick-166148-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2697657Z * [new branch] cherry-pick-166181-by-pytorch_bot_bot_ -> origin/cherry-pick-166181-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2698561Z * [new branch] cherry-pick-166404-by-pytorch_bot_bot_ -> origin/cherry-pick-166404-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2699057Z * [new branch] cherry-pick-166427-by-pytorch_bot_bot_ -> origin/cherry-pick-166427-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2699818Z * [new branch] cherry-pick-166480-by-pytorch_bot_bot_ -> origin/cherry-pick-166480-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2700487Z * [new branch] cherry-pick-166570-by-pytorch_bot_bot_ -> origin/cherry-pick-166570-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2700970Z * [new branch] cherry-pick-166993-by-pytorch_bot_bot_ -> origin/cherry-pick-166993-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2706010Z * [new branch] cherry-pick-167111-by-pytorch_bot_bot_ -> origin/cherry-pick-167111-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2706816Z * [new branch] cherry-pick-167478-by-pytorch_bot_bot_ -> origin/cherry-pick-167478-by-pytorch_bot_bot_ 2025-12-04T09:20:54.2707271Z * [new branch] cherry_pick_166036_166040 -> origin/cherry_pick_166036_166040 2025-12-04T09:20:54.2707638Z * [new branch] cherry_pick_166457 -> origin/cherry_pick_166457 2025-12-04T09:20:54.2707997Z * [new branch] cherrypick_166338 -> origin/cherrypick_166338 2025-12-04T09:20:54.2708313Z * [new branch] cherrypick_166458 -> origin/cherrypick_166458 2025-12-04T09:20:54.2708616Z * [new branch] cherrypick_166586 -> origin/cherrypick_166586 2025-12-04T09:20:54.2708915Z * [new branch] cherrypick_166956 -> origin/cherrypick_166956 2025-12-04T09:20:54.2709196Z * [new branch] ci_attn -> origin/ci_attn 2025-12-04T09:20:54.2709530Z * [new branch] codex-testing -> origin/codex-testing 2025-12-04T09:20:54.2709977Z * [new branch] codex/add-check_memory_overlap-helper-functions -> origin/codex/add-check_memory_overlap-helper-functions 2025-12-04T09:20:54.2714200Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-12-04T09:20:54.2714720Z * [new branch] codex/investigate-segfaults-in-get_tensor_storage_id -> origin/codex/investigate-segfaults-in-get_tensor_storage_id 2025-12-04T09:20:54.2715338Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-12-04T09:20:54.2715794Z * [new branch] compatiblpy39util -> origin/compatiblpy39util 2025-12-04T09:20:54.2716116Z * [new branch] cond_hop_device -> origin/cond_hop_device 2025-12-04T09:20:54.2716406Z * [new branch] context_test -> origin/context_test 2025-12-04T09:20:54.2718861Z * [new branch] copilot/code-style-cleanup-python-pip -> origin/copilot/code-style-cleanup-python-pip 2025-12-04T09:20:54.2719288Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-12-04T09:20:54.2719656Z * [new branch] cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade 2025-12-04T09:20:54.2720069Z * [new branch] crpa/typo-in-inductor_comm_lowering -> origin/crpa/typo-in-inductor_comm_lowering 2025-12-04T09:20:54.2720475Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-12-04T09:20:54.2720884Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-12-04T09:20:54.2721237Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-12-04T09:20:54.2722851Z * [new branch] csl/clean_up -> origin/csl/clean_up 2025-12-04T09:20:54.2723203Z * [new branch] csl/fix_retry_segfault_exit -> origin/csl/fix_retry_segfault_exit 2025-12-04T09:20:54.2723530Z * [new branch] csl/katex -> origin/csl/katex 2025-12-04T09:20:54.2723967Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-12-04T09:20:54.2724277Z * [new branch] csl/lint_testing -> origin/csl/lint_testing 2025-12-04T09:20:54.2724584Z * [new branch] csl/lint_thing -> origin/csl/lint_thing 2025-12-04T09:20:54.2724974Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-12-04T09:20:54.2727070Z * [new branch] csl/manually_gen_json -> origin/csl/manually_gen_json 2025-12-04T09:20:54.2727422Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-12-04T09:20:54.2727773Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-12-04T09:20:54.2728114Z * [new branch] csl/print_timing -> origin/csl/print_timing 2025-12-04T09:20:54.2728451Z * [new branch] csl/remove_experiment -> origin/csl/remove_experiment 2025-12-04T09:20:54.2728811Z * [new branch] csl/remove_maybe_unused_var -> origin/csl/remove_maybe_unused_var 2025-12-04T09:20:54.2729215Z * [new branch] csl/remove_repo_specific_autolabel -> origin/csl/remove_repo_specific_autolabel 2025-12-04T09:20:54.2731831Z * [new branch] csl/remove_run_parallel -> origin/csl/remove_run_parallel 2025-12-04T09:20:54.2732227Z * [new branch] csl/remove_unused_vars -> origin/csl/remove_unused_vars 2025-12-04T09:20:54.2735910Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-12-04T09:20:54.2736438Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-12-04T09:20:54.2740199Z * [new branch] csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs 2025-12-04T09:20:54.2740630Z * [new branch] csl/td_job_level -> origin/csl/td_job_level 2025-12-04T09:20:54.2741049Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-12-04T09:20:54.2741471Z * [new branch] csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn 2025-12-04T09:20:54.2741891Z * [new branch] csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence 2025-12-04T09:20:54.2742282Z * [new branch] csl/upload_json_running -> origin/csl/upload_json_running 2025-12-04T09:20:54.2742669Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-12-04T09:20:54.2742979Z * [new branch] csl/xml_stuff -> origin/csl/xml_stuff 2025-12-04T09:20:54.2743281Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-12-04T09:20:54.2743580Z * [new branch] cuda_mempool -> origin/cuda_mempool 2025-12-04T09:20:54.2743890Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-12-04T09:20:54.2744248Z * [new branch] d4l3k/debug_plane_frtrace -> origin/d4l3k/debug_plane_frtrace 2025-12-04T09:20:54.2744925Z * [new branch] daxia6/2.8o3 -> origin/daxia6/2.8o3 2025-12-04T09:20:54.2745224Z * [new branch] debug-guard -> origin/debug-guard 2025-12-04T09:20:54.2745541Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-12-04T09:20:54.2746109Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 2025-12-04T09:20:54.2746909Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 2025-12-04T09:20:54.2748848Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-12-04T09:20:54.2749544Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-12-04T09:20:54.2750090Z * [new branch] dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt 2025-12-04T09:20:54.2750786Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-12-04T09:20:54.2751192Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-12-04T09:20:54.2752298Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-12-04T09:20:54.2752628Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-12-04T09:20:54.2752978Z * [new branch] dev/joona/fix_sdpa_memtest -> origin/dev/joona/fix_sdpa_memtest 2025-12-04T09:20:54.2753339Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-12-04T09:20:54.2753705Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-12-04T09:20:54.2754054Z * [new branch] dev/joona/scalar_clamp -> origin/dev/joona/scalar_clamp 2025-12-04T09:20:54.2756549Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-12-04T09:20:54.2756884Z * [new branch] dev/joona/sdpa_api -> origin/dev/joona/sdpa_api 2025-12-04T09:20:54.2757217Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-12-04T09:20:54.2757562Z * [new branch] dev/joona/ulpAssertClose -> origin/dev/joona/ulpAssertClose 2025-12-04T09:20:54.2757907Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-12-04T09:20:54.2758214Z * [new branch] disp_counter -> origin/disp_counter 2025-12-04T09:20:54.2761261Z * [new branch] divyanshk-patch-1 -> origin/divyanshk-patch-1 2025-12-04T09:20:54.2761576Z * [new branch] docs -> origin/docs 2025-12-04T09:20:54.2761884Z * [new branch] documentation -> origin/documentation 2025-12-04T09:20:54.2762215Z * [new branch] eager_model_benchmarks -> origin/eager_model_benchmarks 2025-12-04T09:20:54.2762575Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-12-04T09:20:54.2762963Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-12-04T09:20:54.2763325Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-12-04T09:20:54.2765523Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-12-04T09:20:54.2765821Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-12-04T09:20:54.2766114Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-12-04T09:20:54.2766413Z * [new branch] eqy-patch-4 -> origin/eqy-patch-4 2025-12-04T09:20:54.2766704Z * [new branch] eqy-patch-5 -> origin/eqy-patch-5 2025-12-04T09:20:54.2766982Z * [new branch] eqy-patch-6 -> origin/eqy-patch-6 2025-12-04T09:20:54.2767301Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-12-04T09:20:54.2767715Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-12-04T09:20:54.2768153Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-12-04T09:20:54.2769002Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-12-04T09:20:54.2769554Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-12-04T09:20:54.2770508Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-12-04T09:20:54.2771056Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-12-04T09:20:54.2771832Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-12-04T09:20:54.2772756Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-12-04T09:20:54.2773407Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-12-04T09:20:54.2774079Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-12-04T09:20:54.2774907Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-12-04T09:20:54.2775696Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-12-04T09:20:54.2776311Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-12-04T09:20:54.2777234Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-12-04T09:20:54.2777709Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-12-04T09:20:54.2778772Z * [new branch] exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization 2025-12-04T09:20:54.2779678Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-12-04T09:20:54.2780467Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-12-04T09:20:54.2781213Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-12-04T09:20:54.2781814Z * [new branch] exec -> origin/exec 2025-12-04T09:20:54.2782335Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-12-04T09:20:54.2783272Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-12-04T09:20:54.2783930Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-12-04T09:20:54.2784286Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-12-04T09:20:54.2784943Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-12-04T09:20:54.2785537Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-12-04T09:20:54.2788319Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-12-04T09:20:54.2788704Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-12-04T09:20:54.2789018Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-12-04T09:20:54.2789352Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-12-04T09:20:54.2789659Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-12-04T09:20:54.2789960Z * [new branch] export-D82250826 -> origin/export-D82250826 2025-12-04T09:20:54.2790299Z * [new branch] export-D82253817 -> origin/export-D82253817 2025-12-04T09:20:54.2791332Z * [new branch] export-D83541846 -> origin/export-D83541846 2025-12-04T09:20:54.2792061Z * [new branch] export-D83627170 -> origin/export-D83627170 2025-12-04T09:20:54.2792410Z * [new branch] export-D83766701 -> origin/export-D83766701 2025-12-04T09:20:54.2793166Z * [new branch] export-D83768878 -> origin/export-D83768878 2025-12-04T09:20:54.2793757Z * [new branch] export-D83769447 -> origin/export-D83769447 2025-12-04T09:20:54.2794455Z * [new branch] export-D84089824 -> origin/export-D84089824 2025-12-04T09:20:54.2795118Z * [new branch] export-D84213020 -> origin/export-D84213020 2025-12-04T09:20:54.2796499Z * [new branch] export-D84373821 -> origin/export-D84373821 2025-12-04T09:20:54.2797001Z * [new branch] export-D84612194 -> origin/export-D84612194 2025-12-04T09:20:54.2797679Z * [new branch] export-D84890985 -> origin/export-D84890985 2025-12-04T09:20:54.2798404Z * [new branch] export-D85122326 -> origin/export-D85122326 2025-12-04T09:20:54.2799728Z * [new branch] export-D86256198 -> origin/export-D86256198 2025-12-04T09:20:54.2800138Z * [new branch] export-D86460608 -> origin/export-D86460608 2025-12-04T09:20:54.2801158Z * [new branch] export-D86474796 -> origin/export-D86474796 2025-12-04T09:20:54.2801748Z * [new branch] export-D86712396 -> origin/export-D86712396 2025-12-04T09:20:54.2802392Z * [new branch] export-D87022129 -> origin/export-D87022129 2025-12-04T09:20:54.2803348Z * [new branch] export-D87838959 -> origin/export-D87838959 2025-12-04T09:20:54.2803934Z * [new branch] export-D88319437 -> origin/export-D88319437 2025-12-04T09:20:54.2804905Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-12-04T09:20:54.2805517Z * [new branch] ezyang-titan-october -> origin/ezyang-titan-october 2025-12-04T09:20:54.2806161Z * [new branch] ezyang-titan-october2 -> origin/ezyang-titan-october2 2025-12-04T09:20:54.2806738Z * [new branch] ezyang-war -> origin/ezyang-war 2025-12-04T09:20:54.2807949Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-12-04T09:20:54.2808923Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-12-04T09:20:54.2809988Z * [new branch] fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm 2025-12-04T09:20:54.2810421Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-12-04T09:20:54.2813329Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-12-04T09:20:54.2813702Z * [new branch] fca -> origin/fca 2025-12-04T09:20:54.2814014Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-12-04T09:20:54.2814318Z * [new branch] fca5 -> origin/fca5 2025-12-04T09:20:54.2814931Z * [new branch] feature/justknobs-cpp -> origin/feature/justknobs-cpp 2025-12-04T09:20:54.2815700Z * [new branch] feature/numa-forkserver -> origin/feature/numa-forkserver 2025-12-04T09:20:54.2819391Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-12-04T09:20:54.2819772Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-12-04T09:20:54.2820107Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-12-04T09:20:54.2820464Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-12-04T09:20:54.2820815Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-12-04T09:20:54.2821159Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-12-04T09:20:54.2821528Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-12-04T09:20:54.2821907Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-12-04T09:20:54.2822640Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-12-04T09:20:54.2823202Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-12-04T09:20:54.2824023Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-12-04T09:20:54.2824768Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-12-04T09:20:54.2825460Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-12-04T09:20:54.2826219Z * [new branch] fix_addmm_issue -> origin/fix_addmm_issue 2025-12-04T09:20:54.2826946Z * [new branch] fix_amd_missing_cluster_dims -> origin/fix_amd_missing_cluster_dims 2025-12-04T09:20:54.2827602Z * [new branch] fix_bench_bwd_pass -> origin/fix_bench_bwd_pass 2025-12-04T09:20:54.2828357Z * [new branch] fix_mem_profiler_config -> origin/fix_mem_profiler_config 2025-12-04T09:20:54.2828974Z * [new branch] fix_nvrtc_discovery -> origin/fix_nvrtc_discovery 2025-12-04T09:20:54.2829653Z * [new branch] fix_op_runner -> origin/fix_op_runner 2025-12-04T09:20:54.2830339Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-12-04T09:20:54.2831079Z * [new branch] fixes-triage -> origin/fixes-triage 2025-12-04T09:20:54.2831753Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-12-04T09:20:54.2833407Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-12-04T09:20:54.2833851Z * [new branch] flex-flash -> origin/flex-flash 2025-12-04T09:20:54.2834198Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-12-04T09:20:54.2834551Z * [new branch] flex_flash -> origin/flex_flash 2025-12-04T09:20:54.2836465Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-12-04T09:20:54.2837026Z * [new branch] fmassa/tests_comm_compute_scheduler -> origin/fmassa/tests_comm_compute_scheduler 2025-12-04T09:20:54.2837467Z * [new branch] forkserver_fix -> origin/forkserver_fix 2025-12-04T09:20:54.2837836Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-12-04T09:20:54.2838252Z * [new branch] fx_cpp -> origin/fx_cpp 2025-12-04T09:20:54.2843087Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-12-04T09:20:54.2843457Z * [new branch] galv-patch-1 -> origin/galv-patch-1 2025-12-04T09:20:54.2843860Z * [new branch] galv/cudagraphs-conditional-nodes-4 -> origin/galv/cudagraphs-conditional-nodes-4 2025-12-04T09:20:54.2844297Z * [new branch] georgehong/cmakelists-patch -> origin/georgehong/cmakelists-patch 2025-12-04T09:20:54.2844693Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-12-04T09:20:54.2845012Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-12-04T09:20:54.2845764Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-12-04T09:20:54.2846403Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-12-04T09:20:54.2848087Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-12-04T09:20:54.2848485Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-12-04T09:20:54.2849867Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-12-04T09:20:54.2850368Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-12-04T09:20:54.2851022Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-12-04T09:20:54.2855550Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-12-04T09:20:54.2855911Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-12-04T09:20:54.2856297Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-12-04T09:20:54.2860542Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-12-04T09:20:54.2860919Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-12-04T09:20:54.2866436Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-12-04T09:20:54.2870474Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-12-04T09:20:54.2874657Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-12-04T09:20:54.2880524Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-12-04T09:20:54.2885793Z * [new branch] gh/H-Huang/226/base -> origin/gh/H-Huang/226/base 2025-12-04T09:20:54.2887932Z * [new branch] gh/H-Huang/226/head -> origin/gh/H-Huang/226/head 2025-12-04T09:20:54.2888347Z * [new branch] gh/H-Huang/226/orig -> origin/gh/H-Huang/226/orig 2025-12-04T09:20:54.2888682Z * [new branch] gh/H-Huang/228/base -> origin/gh/H-Huang/228/base 2025-12-04T09:20:54.2888990Z * [new branch] gh/H-Huang/228/head -> origin/gh/H-Huang/228/head 2025-12-04T09:20:54.2889301Z * [new branch] gh/H-Huang/228/orig -> origin/gh/H-Huang/228/orig 2025-12-04T09:20:54.2889644Z * [new branch] gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base 2025-12-04T09:20:54.2890008Z * [new branch] gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head 2025-12-04T09:20:54.2890389Z * [new branch] gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig 2025-12-04T09:20:54.2890742Z * [new branch] gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base 2025-12-04T09:20:54.2891077Z * [new branch] gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head 2025-12-04T09:20:54.2891419Z * [new branch] gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig 2025-12-04T09:20:54.2892002Z * [new branch] gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base 2025-12-04T09:20:54.2892367Z * [new branch] gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head 2025-12-04T09:20:54.2892726Z * [new branch] gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig 2025-12-04T09:20:54.2893079Z * [new branch] gh/IvanKobzarev/162/base -> origin/gh/IvanKobzarev/162/base 2025-12-04T09:20:54.2893435Z * [new branch] gh/IvanKobzarev/162/head -> origin/gh/IvanKobzarev/162/head 2025-12-04T09:20:54.2893809Z * [new branch] gh/IvanKobzarev/162/orig -> origin/gh/IvanKobzarev/162/orig 2025-12-04T09:20:54.2894141Z * [new branch] gh/IvanKobzarev/163/base -> origin/gh/IvanKobzarev/163/base 2025-12-04T09:20:54.2894464Z * [new branch] gh/IvanKobzarev/163/head -> origin/gh/IvanKobzarev/163/head 2025-12-04T09:20:54.2894833Z * [new branch] gh/IvanKobzarev/163/orig -> origin/gh/IvanKobzarev/163/orig 2025-12-04T09:20:54.2895154Z * [new branch] gh/IvanKobzarev/166/base -> origin/gh/IvanKobzarev/166/base 2025-12-04T09:20:54.2895510Z * [new branch] gh/IvanKobzarev/166/head -> origin/gh/IvanKobzarev/166/head 2025-12-04T09:20:54.2895835Z * [new branch] gh/IvanKobzarev/166/orig -> origin/gh/IvanKobzarev/166/orig 2025-12-04T09:20:54.2896166Z * [new branch] gh/IvanKobzarev/167/base -> origin/gh/IvanKobzarev/167/base 2025-12-04T09:20:54.2896710Z * [new branch] gh/IvanKobzarev/167/head -> origin/gh/IvanKobzarev/167/head 2025-12-04T09:20:54.2897044Z * [new branch] gh/IvanKobzarev/167/orig -> origin/gh/IvanKobzarev/167/orig 2025-12-04T09:20:54.2897371Z * [new branch] gh/IvanKobzarev/168/base -> origin/gh/IvanKobzarev/168/base 2025-12-04T09:20:54.2897757Z * [new branch] gh/IvanKobzarev/168/head -> origin/gh/IvanKobzarev/168/head 2025-12-04T09:20:54.2898085Z * [new branch] gh/IvanKobzarev/168/orig -> origin/gh/IvanKobzarev/168/orig 2025-12-04T09:20:54.2898404Z * [new branch] gh/IvanKobzarev/169/base -> origin/gh/IvanKobzarev/169/base 2025-12-04T09:20:54.2898727Z * [new branch] gh/IvanKobzarev/169/head -> origin/gh/IvanKobzarev/169/head 2025-12-04T09:20:54.2899052Z * [new branch] gh/IvanKobzarev/169/orig -> origin/gh/IvanKobzarev/169/orig 2025-12-04T09:20:54.2899376Z * [new branch] gh/IvanKobzarev/170/base -> origin/gh/IvanKobzarev/170/base 2025-12-04T09:20:54.2899697Z * [new branch] gh/IvanKobzarev/170/head -> origin/gh/IvanKobzarev/170/head 2025-12-04T09:20:54.2900027Z * [new branch] gh/IvanKobzarev/170/orig -> origin/gh/IvanKobzarev/170/orig 2025-12-04T09:20:54.2900354Z * [new branch] gh/IvanKobzarev/171/base -> origin/gh/IvanKobzarev/171/base 2025-12-04T09:20:54.2900683Z * [new branch] gh/IvanKobzarev/171/head -> origin/gh/IvanKobzarev/171/head 2025-12-04T09:20:54.2901003Z * [new branch] gh/IvanKobzarev/171/orig -> origin/gh/IvanKobzarev/171/orig 2025-12-04T09:20:54.2901331Z * [new branch] gh/IvanKobzarev/172/base -> origin/gh/IvanKobzarev/172/base 2025-12-04T09:20:54.2901656Z * [new branch] gh/IvanKobzarev/172/head -> origin/gh/IvanKobzarev/172/head 2025-12-04T09:20:54.2901974Z * [new branch] gh/IvanKobzarev/172/orig -> origin/gh/IvanKobzarev/172/orig 2025-12-04T09:20:54.2902302Z * [new branch] gh/IvanKobzarev/173/base -> origin/gh/IvanKobzarev/173/base 2025-12-04T09:20:54.2902641Z * [new branch] gh/IvanKobzarev/173/head -> origin/gh/IvanKobzarev/173/head 2025-12-04T09:20:54.2902964Z * [new branch] gh/IvanKobzarev/173/orig -> origin/gh/IvanKobzarev/173/orig 2025-12-04T09:20:54.2903286Z * [new branch] gh/IvanKobzarev/174/base -> origin/gh/IvanKobzarev/174/base 2025-12-04T09:20:54.2903612Z * [new branch] gh/IvanKobzarev/174/head -> origin/gh/IvanKobzarev/174/head 2025-12-04T09:20:54.2903937Z * [new branch] gh/IvanKobzarev/174/orig -> origin/gh/IvanKobzarev/174/orig 2025-12-04T09:20:54.2904261Z * [new branch] gh/IvanKobzarev/175/base -> origin/gh/IvanKobzarev/175/base 2025-12-04T09:20:54.2904579Z * [new branch] gh/IvanKobzarev/175/head -> origin/gh/IvanKobzarev/175/head 2025-12-04T09:20:54.2904904Z * [new branch] gh/IvanKobzarev/175/orig -> origin/gh/IvanKobzarev/175/orig 2025-12-04T09:20:54.2905240Z * [new branch] gh/IvanKobzarev/176/base -> origin/gh/IvanKobzarev/176/base 2025-12-04T09:20:54.2905559Z * [new branch] gh/IvanKobzarev/176/head -> origin/gh/IvanKobzarev/176/head 2025-12-04T09:20:54.2905878Z * [new branch] gh/IvanKobzarev/176/orig -> origin/gh/IvanKobzarev/176/orig 2025-12-04T09:20:54.2906198Z * [new branch] gh/IvanKobzarev/177/base -> origin/gh/IvanKobzarev/177/base 2025-12-04T09:20:54.2906523Z * [new branch] gh/IvanKobzarev/177/head -> origin/gh/IvanKobzarev/177/head 2025-12-04T09:20:54.2908035Z * [new branch] gh/IvanKobzarev/177/orig -> origin/gh/IvanKobzarev/177/orig 2025-12-04T09:20:54.2908498Z * [new branch] gh/IvanKobzarev/178/base -> origin/gh/IvanKobzarev/178/base 2025-12-04T09:20:54.2913222Z * [new branch] gh/IvanKobzarev/178/head -> origin/gh/IvanKobzarev/178/head 2025-12-04T09:20:54.2917305Z * [new branch] gh/IvanKobzarev/178/orig -> origin/gh/IvanKobzarev/178/orig 2025-12-04T09:20:54.2919149Z * [new branch] gh/IvanKobzarev/179/base -> origin/gh/IvanKobzarev/179/base 2025-12-04T09:20:54.2919569Z * [new branch] gh/IvanKobzarev/179/head -> origin/gh/IvanKobzarev/179/head 2025-12-04T09:20:54.2920223Z * [new branch] gh/IvanKobzarev/179/orig -> origin/gh/IvanKobzarev/179/orig 2025-12-04T09:20:54.2920633Z * [new branch] gh/IvanKobzarev/180/base -> origin/gh/IvanKobzarev/180/base 2025-12-04T09:20:54.2920968Z * [new branch] gh/IvanKobzarev/180/head -> origin/gh/IvanKobzarev/180/head 2025-12-04T09:20:54.2921304Z * [new branch] gh/IvanKobzarev/180/orig -> origin/gh/IvanKobzarev/180/orig 2025-12-04T09:20:54.2921678Z * [new branch] gh/IvanKobzarev/181/base -> origin/gh/IvanKobzarev/181/base 2025-12-04T09:20:54.2922053Z * [new branch] gh/IvanKobzarev/181/head -> origin/gh/IvanKobzarev/181/head 2025-12-04T09:20:54.2922396Z * [new branch] gh/IvanKobzarev/181/orig -> origin/gh/IvanKobzarev/181/orig 2025-12-04T09:20:54.2922738Z * [new branch] gh/IvanKobzarev/182/base -> origin/gh/IvanKobzarev/182/base 2025-12-04T09:20:54.2923082Z * [new branch] gh/IvanKobzarev/182/head -> origin/gh/IvanKobzarev/182/head 2025-12-04T09:20:54.2923428Z * [new branch] gh/IvanKobzarev/182/orig -> origin/gh/IvanKobzarev/182/orig 2025-12-04T09:20:54.2923780Z * [new branch] gh/IvanKobzarev/183/base -> origin/gh/IvanKobzarev/183/base 2025-12-04T09:20:54.2924131Z * [new branch] gh/IvanKobzarev/183/head -> origin/gh/IvanKobzarev/183/head 2025-12-04T09:20:54.2924473Z * [new branch] gh/IvanKobzarev/183/orig -> origin/gh/IvanKobzarev/183/orig 2025-12-04T09:20:54.2924827Z * [new branch] gh/IvanKobzarev/184/base -> origin/gh/IvanKobzarev/184/base 2025-12-04T09:20:54.2925171Z * [new branch] gh/IvanKobzarev/184/head -> origin/gh/IvanKobzarev/184/head 2025-12-04T09:20:54.2925513Z * [new branch] gh/IvanKobzarev/184/orig -> origin/gh/IvanKobzarev/184/orig 2025-12-04T09:20:54.2925862Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-12-04T09:20:54.2926200Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-12-04T09:20:54.2926531Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-12-04T09:20:54.2926876Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-12-04T09:20:54.2927215Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-12-04T09:20:54.2927545Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-12-04T09:20:54.2927880Z * [new branch] gh/NikhilAPatel/5/base -> origin/gh/NikhilAPatel/5/base 2025-12-04T09:20:54.2928221Z * [new branch] gh/NikhilAPatel/5/head -> origin/gh/NikhilAPatel/5/head 2025-12-04T09:20:54.2928556Z * [new branch] gh/NikhilAPatel/5/orig -> origin/gh/NikhilAPatel/5/orig 2025-12-04T09:20:54.2928957Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-12-04T09:20:54.2929279Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-12-04T09:20:54.2929591Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-12-04T09:20:54.2929897Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-12-04T09:20:54.2930190Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-12-04T09:20:54.2930491Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-12-04T09:20:54.2930889Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-12-04T09:20:54.2931197Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-12-04T09:20:54.2931680Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-12-04T09:20:54.2932112Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-12-04T09:20:54.2932524Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-12-04T09:20:54.2932830Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-12-04T09:20:54.2933172Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-12-04T09:20:54.2933472Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-12-04T09:20:54.2933770Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-12-04T09:20:54.2934060Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-12-04T09:20:54.2934363Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-12-04T09:20:54.2934874Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-12-04T09:20:54.2935677Z * [new branch] gh/PaliC/25/head -> origin/gh/PaliC/25/head 2025-12-04T09:20:54.2941636Z * [new branch] gh/PaliC/25/next -> origin/gh/PaliC/25/next 2025-12-04T09:20:54.2942027Z * [new branch] gh/PaliC/25/orig -> origin/gh/PaliC/25/orig 2025-12-04T09:20:54.2942348Z * [new branch] gh/PaliC/26/head -> origin/gh/PaliC/26/head 2025-12-04T09:20:54.2942731Z * [new branch] gh/PaliC/26/next -> origin/gh/PaliC/26/next 2025-12-04T09:20:54.2943040Z * [new branch] gh/PaliC/26/orig -> origin/gh/PaliC/26/orig 2025-12-04T09:20:54.2943336Z * [new branch] gh/PaliC/27/next -> origin/gh/PaliC/27/next 2025-12-04T09:20:54.2943663Z * [new branch] gh/PaliC/28/head -> origin/gh/PaliC/28/head 2025-12-04T09:20:54.2943971Z * [new branch] gh/PaliC/28/next -> origin/gh/PaliC/28/next 2025-12-04T09:20:54.2944266Z * [new branch] gh/PaliC/28/orig -> origin/gh/PaliC/28/orig 2025-12-04T09:20:54.2944575Z * [new branch] gh/PaliC/29/head -> origin/gh/PaliC/29/head 2025-12-04T09:20:54.2944879Z * [new branch] gh/PaliC/29/next -> origin/gh/PaliC/29/next 2025-12-04T09:20:54.2945393Z * [new branch] gh/PaliC/29/orig -> origin/gh/PaliC/29/orig 2025-12-04T09:20:54.2945697Z * [new branch] gh/PaliC/30/head -> origin/gh/PaliC/30/head 2025-12-04T09:20:54.2945998Z * [new branch] gh/PaliC/30/next -> origin/gh/PaliC/30/next 2025-12-04T09:20:54.2946303Z * [new branch] gh/PaliC/30/orig -> origin/gh/PaliC/30/orig 2025-12-04T09:20:54.2946736Z * [new branch] gh/PaliC/31/head -> origin/gh/PaliC/31/head 2025-12-04T09:20:54.2952083Z * [new branch] gh/PaliC/31/next -> origin/gh/PaliC/31/next 2025-12-04T09:20:54.2953897Z * [new branch] gh/PaliC/31/orig -> origin/gh/PaliC/31/orig 2025-12-04T09:20:54.2954312Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-12-04T09:20:54.2954678Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-12-04T09:20:54.2955065Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-12-04T09:20:54.2955423Z * [new branch] gh/PaulZhang12/28/base -> origin/gh/PaulZhang12/28/base 2025-12-04T09:20:54.2955788Z * [new branch] gh/PaulZhang12/28/head -> origin/gh/PaulZhang12/28/head 2025-12-04T09:20:54.2956315Z * [new branch] gh/PaulZhang12/28/orig -> origin/gh/PaulZhang12/28/orig 2025-12-04T09:20:54.2956649Z * [new branch] gh/PaulZhang12/31/base -> origin/gh/PaulZhang12/31/base 2025-12-04T09:20:54.2957020Z * [new branch] gh/PaulZhang12/31/head -> origin/gh/PaulZhang12/31/head 2025-12-04T09:20:54.2957447Z * [new branch] gh/PaulZhang12/31/orig -> origin/gh/PaulZhang12/31/orig 2025-12-04T09:20:54.2957783Z * [new branch] gh/PaulZhang12/37/base -> origin/gh/PaulZhang12/37/base 2025-12-04T09:20:54.2958100Z * [new branch] gh/PaulZhang12/37/head -> origin/gh/PaulZhang12/37/head 2025-12-04T09:20:54.2958429Z * [new branch] gh/PaulZhang12/37/orig -> origin/gh/PaulZhang12/37/orig 2025-12-04T09:20:54.2958758Z * [new branch] gh/PaulZhang12/40/base -> origin/gh/PaulZhang12/40/base 2025-12-04T09:20:54.2959087Z * [new branch] gh/PaulZhang12/40/head -> origin/gh/PaulZhang12/40/head 2025-12-04T09:20:54.2962430Z * [new branch] gh/PaulZhang12/40/orig -> origin/gh/PaulZhang12/40/orig 2025-12-04T09:20:54.2962768Z * [new branch] gh/PaulZhang12/42/base -> origin/gh/PaulZhang12/42/base 2025-12-04T09:20:54.2963100Z * [new branch] gh/PaulZhang12/42/head -> origin/gh/PaulZhang12/42/head 2025-12-04T09:20:54.2963430Z * [new branch] gh/PaulZhang12/43/base -> origin/gh/PaulZhang12/43/base 2025-12-04T09:20:54.2963747Z * [new branch] gh/PaulZhang12/43/head -> origin/gh/PaulZhang12/43/head 2025-12-04T09:20:54.2964071Z * [new branch] gh/PaulZhang12/43/orig -> origin/gh/PaulZhang12/43/orig 2025-12-04T09:20:54.2964404Z * [new branch] gh/PaulZhang12/44/base -> origin/gh/PaulZhang12/44/base 2025-12-04T09:20:54.2964729Z * [new branch] gh/PaulZhang12/44/head -> origin/gh/PaulZhang12/44/head 2025-12-04T09:20:54.2965050Z * [new branch] gh/PaulZhang12/45/base -> origin/gh/PaulZhang12/45/base 2025-12-04T09:20:54.2965376Z * [new branch] gh/PaulZhang12/45/head -> origin/gh/PaulZhang12/45/head 2025-12-04T09:20:54.2965872Z * [new branch] gh/PaulZhang12/45/orig -> origin/gh/PaulZhang12/45/orig 2025-12-04T09:20:54.2967050Z * [new branch] gh/PaulZhang12/46/base -> origin/gh/PaulZhang12/46/base 2025-12-04T09:20:54.2967580Z * [new branch] gh/PaulZhang12/46/head -> origin/gh/PaulZhang12/46/head 2025-12-04T09:20:54.2968195Z * [new branch] gh/PaulZhang12/46/orig -> origin/gh/PaulZhang12/46/orig 2025-12-04T09:20:54.2969356Z * [new branch] gh/PaulZhang12/47/base -> origin/gh/PaulZhang12/47/base 2025-12-04T09:20:54.2969883Z * [new branch] gh/PaulZhang12/47/head -> origin/gh/PaulZhang12/47/head 2025-12-04T09:20:54.2970573Z * [new branch] gh/PaulZhang12/47/orig -> origin/gh/PaulZhang12/47/orig 2025-12-04T09:20:54.2971770Z * [new branch] gh/PaulZhang12/48/base -> origin/gh/PaulZhang12/48/base 2025-12-04T09:20:54.2972134Z * [new branch] gh/PaulZhang12/48/head -> origin/gh/PaulZhang12/48/head 2025-12-04T09:20:54.2972894Z * [new branch] gh/PaulZhang12/48/orig -> origin/gh/PaulZhang12/48/orig 2025-12-04T09:20:54.2974367Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-12-04T09:20:54.2974797Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-12-04T09:20:54.2976389Z * [new branch] gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base 2025-12-04T09:20:54.2976755Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2025-12-04T09:20:54.2979770Z * [new branch] gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base 2025-12-04T09:20:54.2980197Z * [new branch] gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head 2025-12-04T09:20:54.2980677Z * [new branch] gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig 2025-12-04T09:20:54.2981087Z * [new branch] gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base 2025-12-04T09:20:54.2985089Z * [new branch] gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head 2025-12-04T09:20:54.2985538Z * [new branch] gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig 2025-12-04T09:20:54.2985939Z * [new branch] gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base 2025-12-04T09:20:54.2991397Z * [new branch] gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head 2025-12-04T09:20:54.2996601Z * [new branch] gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig 2025-12-04T09:20:54.2998645Z * [new branch] gh/SherlockNoMad/15/base -> origin/gh/SherlockNoMad/15/base 2025-12-04T09:20:54.2999177Z * [new branch] gh/SherlockNoMad/15/head -> origin/gh/SherlockNoMad/15/head 2025-12-04T09:20:54.3004096Z * [new branch] gh/SherlockNoMad/15/orig -> origin/gh/SherlockNoMad/15/orig 2025-12-04T09:20:54.3006598Z * [new branch] gh/SherlockNoMad/17/base -> origin/gh/SherlockNoMad/17/base 2025-12-04T09:20:54.3006999Z * [new branch] gh/SherlockNoMad/17/head -> origin/gh/SherlockNoMad/17/head 2025-12-04T09:20:54.3007334Z * [new branch] gh/SherlockNoMad/17/orig -> origin/gh/SherlockNoMad/17/orig 2025-12-04T09:20:54.3007663Z * [new branch] gh/SherlockNoMad/18/base -> origin/gh/SherlockNoMad/18/base 2025-12-04T09:20:54.3007994Z * [new branch] gh/SherlockNoMad/18/head -> origin/gh/SherlockNoMad/18/head 2025-12-04T09:20:54.3008310Z * [new branch] gh/SherlockNoMad/18/orig -> origin/gh/SherlockNoMad/18/orig 2025-12-04T09:20:54.3008631Z * [new branch] gh/SherlockNoMad/19/base -> origin/gh/SherlockNoMad/19/base 2025-12-04T09:20:54.3008984Z * [new branch] gh/SherlockNoMad/19/head -> origin/gh/SherlockNoMad/19/head 2025-12-04T09:20:54.3009320Z * [new branch] gh/SherlockNoMad/19/orig -> origin/gh/SherlockNoMad/19/orig 2025-12-04T09:20:54.3009664Z * [new branch] gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base 2025-12-04T09:20:54.3010024Z * [new branch] gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head 2025-12-04T09:20:54.3010360Z * [new branch] gh/SherlockNoMad/20/base -> origin/gh/SherlockNoMad/20/base 2025-12-04T09:20:54.3010684Z * [new branch] gh/SherlockNoMad/20/head -> origin/gh/SherlockNoMad/20/head 2025-12-04T09:20:54.3011003Z * [new branch] gh/SherlockNoMad/20/orig -> origin/gh/SherlockNoMad/20/orig 2025-12-04T09:20:54.3011334Z * [new branch] gh/SherlockNoMad/21/base -> origin/gh/SherlockNoMad/21/base 2025-12-04T09:20:54.3011896Z * [new branch] gh/SherlockNoMad/21/head -> origin/gh/SherlockNoMad/21/head 2025-12-04T09:20:54.3012267Z * [new branch] gh/SherlockNoMad/21/orig -> origin/gh/SherlockNoMad/21/orig 2025-12-04T09:20:54.3012643Z * [new branch] gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base 2025-12-04T09:20:54.3013024Z * [new branch] gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head 2025-12-04T09:20:54.3013381Z * [new branch] gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base 2025-12-04T09:20:54.3013712Z * [new branch] gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head 2025-12-04T09:20:54.3014049Z * [new branch] gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base 2025-12-04T09:20:54.3014388Z * [new branch] gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head 2025-12-04T09:20:54.3014753Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-12-04T09:20:54.3015240Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-12-04T09:20:54.3015590Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-12-04T09:20:54.3015934Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-12-04T09:20:54.3016323Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-12-04T09:20:54.3016655Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-12-04T09:20:54.3016997Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-12-04T09:20:54.3017322Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-12-04T09:20:54.3017645Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-12-04T09:20:54.3017962Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-12-04T09:20:54.3018294Z * [new branch] gh/StrongerXi/73/base -> origin/gh/StrongerXi/73/base 2025-12-04T09:20:54.3018626Z * [new branch] gh/StrongerXi/73/head -> origin/gh/StrongerXi/73/head 2025-12-04T09:20:54.3019148Z * [new branch] gh/StrongerXi/73/orig -> origin/gh/StrongerXi/73/orig 2025-12-04T09:20:54.3019564Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-12-04T09:20:54.3024872Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-12-04T09:20:54.3027724Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-12-04T09:20:54.3028110Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-12-04T09:20:54.3028433Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-12-04T09:20:54.3028757Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-12-04T09:20:54.3029070Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-12-04T09:20:54.3029377Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-12-04T09:20:54.3029691Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-12-04T09:20:54.3029996Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-12-04T09:20:54.3030302Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-12-04T09:20:54.3030607Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-12-04T09:20:54.3030905Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-12-04T09:20:54.3031212Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-12-04T09:20:54.3031524Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-12-04T09:20:54.3031827Z * [new branch] gh/XilunWu/171/base -> origin/gh/XilunWu/171/base 2025-12-04T09:20:54.3032126Z * [new branch] gh/XilunWu/171/head -> origin/gh/XilunWu/171/head 2025-12-04T09:20:54.3032436Z * [new branch] gh/XilunWu/171/orig -> origin/gh/XilunWu/171/orig 2025-12-04T09:20:54.3032784Z * [new branch] gh/XilunWu/173/base -> origin/gh/XilunWu/173/base 2025-12-04T09:20:54.3033098Z * [new branch] gh/XilunWu/173/head -> origin/gh/XilunWu/173/head 2025-12-04T09:20:54.3033399Z * [new branch] gh/XilunWu/173/orig -> origin/gh/XilunWu/173/orig 2025-12-04T09:20:54.3033704Z * [new branch] gh/XilunWu/175/base -> origin/gh/XilunWu/175/base 2025-12-04T09:20:54.3034012Z * [new branch] gh/XilunWu/175/head -> origin/gh/XilunWu/175/head 2025-12-04T09:20:54.3034453Z * [new branch] gh/XilunWu/175/orig -> origin/gh/XilunWu/175/orig 2025-12-04T09:20:54.3034753Z * [new branch] gh/XilunWu/176/base -> origin/gh/XilunWu/176/base 2025-12-04T09:20:54.3035380Z * [new branch] gh/XilunWu/176/head -> origin/gh/XilunWu/176/head 2025-12-04T09:20:54.3035861Z * [new branch] gh/XilunWu/176/orig -> origin/gh/XilunWu/176/orig 2025-12-04T09:20:54.3036210Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-12-04T09:20:54.3036553Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-12-04T09:20:54.3036880Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-12-04T09:20:54.3039686Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-12-04T09:20:54.3040045Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-12-04T09:20:54.3040373Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-12-04T09:20:54.3040697Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-12-04T09:20:54.3041012Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-12-04T09:20:54.3041330Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-12-04T09:20:54.3045635Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-12-04T09:20:54.3047413Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-12-04T09:20:54.3047774Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-12-04T09:20:54.3048093Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-12-04T09:20:54.3048445Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-12-04T09:20:54.3048768Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-12-04T09:20:54.3049090Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-12-04T09:20:54.3049421Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-12-04T09:20:54.3049753Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-12-04T09:20:54.3050074Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-12-04T09:20:54.3050394Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-12-04T09:20:54.3050711Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-12-04T09:20:54.3051031Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-12-04T09:20:54.3051367Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-12-04T09:20:54.3052846Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-12-04T09:20:54.3053211Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-12-04T09:20:54.3053573Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-12-04T09:20:54.3053904Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-12-04T09:20:54.3054245Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-12-04T09:20:54.3054569Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-12-04T09:20:54.3054908Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-12-04T09:20:54.3058295Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-12-04T09:20:54.3058770Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-12-04T09:20:54.3059079Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-12-04T09:20:54.3059463Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-12-04T09:20:54.3059782Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-12-04T09:20:54.3060104Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-12-04T09:20:54.3060411Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-12-04T09:20:54.3060724Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-12-04T09:20:54.3062370Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-12-04T09:20:54.3062691Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-12-04T09:20:54.3062995Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-12-04T09:20:54.3063304Z * [new branch] gh/XuehaiPan/390/base -> origin/gh/XuehaiPan/390/base 2025-12-04T09:20:54.3063618Z * [new branch] gh/XuehaiPan/390/head -> origin/gh/XuehaiPan/390/head 2025-12-04T09:20:54.3063918Z * [new branch] gh/XuehaiPan/390/orig -> origin/gh/XuehaiPan/390/orig 2025-12-04T09:20:54.3067153Z * [new branch] gh/XuehaiPan/391/base -> origin/gh/XuehaiPan/391/base 2025-12-04T09:20:54.3067467Z * [new branch] gh/XuehaiPan/391/head -> origin/gh/XuehaiPan/391/head 2025-12-04T09:20:54.3067778Z * [new branch] gh/XuehaiPan/391/orig -> origin/gh/XuehaiPan/391/orig 2025-12-04T09:20:54.3068227Z * [new branch] gh/XuehaiPan/392/base -> origin/gh/XuehaiPan/392/base 2025-12-04T09:20:54.3068562Z * [new branch] gh/XuehaiPan/392/head -> origin/gh/XuehaiPan/392/head 2025-12-04T09:20:54.3068878Z * [new branch] gh/XuehaiPan/392/orig -> origin/gh/XuehaiPan/392/orig 2025-12-04T09:20:54.3071519Z * [new branch] gh/XuehaiPan/394/base -> origin/gh/XuehaiPan/394/base 2025-12-04T09:20:54.3071857Z * [new branch] gh/XuehaiPan/394/head -> origin/gh/XuehaiPan/394/head 2025-12-04T09:20:54.3072189Z * [new branch] gh/XuehaiPan/394/orig -> origin/gh/XuehaiPan/394/orig 2025-12-04T09:20:54.3072705Z * [new branch] gh/XuehaiPan/397/base -> origin/gh/XuehaiPan/397/base 2025-12-04T09:20:54.3073031Z * [new branch] gh/XuehaiPan/397/head -> origin/gh/XuehaiPan/397/head 2025-12-04T09:20:54.3073354Z * [new branch] gh/XuehaiPan/397/orig -> origin/gh/XuehaiPan/397/orig 2025-12-04T09:20:54.3073665Z * [new branch] gh/XuehaiPan/398/base -> origin/gh/XuehaiPan/398/base 2025-12-04T09:20:54.3079143Z * [new branch] gh/XuehaiPan/398/head -> origin/gh/XuehaiPan/398/head 2025-12-04T09:20:54.3083900Z * [new branch] gh/XuehaiPan/398/orig -> origin/gh/XuehaiPan/398/orig 2025-12-04T09:20:54.3089530Z * [new branch] gh/XuehaiPan/399/base -> origin/gh/XuehaiPan/399/base 2025-12-04T09:20:54.3089954Z * [new branch] gh/XuehaiPan/399/head -> origin/gh/XuehaiPan/399/head 2025-12-04T09:20:54.3090304Z * [new branch] gh/XuehaiPan/399/orig -> origin/gh/XuehaiPan/399/orig 2025-12-04T09:20:54.3090633Z * [new branch] gh/XuehaiPan/400/base -> origin/gh/XuehaiPan/400/base 2025-12-04T09:20:54.3090968Z * [new branch] gh/XuehaiPan/400/head -> origin/gh/XuehaiPan/400/head 2025-12-04T09:20:54.3091303Z * [new branch] gh/XuehaiPan/400/orig -> origin/gh/XuehaiPan/400/orig 2025-12-04T09:20:54.3092054Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-12-04T09:20:54.3092441Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-12-04T09:20:54.3092822Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-12-04T09:20:54.3093244Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-12-04T09:20:54.3093592Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-12-04T09:20:54.3093952Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-12-04T09:20:54.3094299Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-12-04T09:20:54.3094623Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-12-04T09:20:54.3094939Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-12-04T09:20:54.3095265Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-12-04T09:20:54.3095591Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-12-04T09:20:54.3095919Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-12-04T09:20:54.3096242Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-12-04T09:20:54.3096559Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-12-04T09:20:54.3096879Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-12-04T09:20:54.3097195Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-12-04T09:20:54.3097515Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-12-04T09:20:54.3097839Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-12-04T09:20:54.3098171Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-12-04T09:20:54.3098695Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-12-04T09:20:54.3099100Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-12-04T09:20:54.3099488Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-12-04T09:20:54.3099924Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-12-04T09:20:54.3100803Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-12-04T09:20:54.3101415Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-12-04T09:20:54.3101916Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2025-12-04T09:20:54.3102421Z * [new branch] gh/albanD/4/head -> origin/gh/albanD/4/head 2025-12-04T09:20:54.3102767Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2025-12-04T09:20:54.3103272Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-12-04T09:20:54.3103741Z * [new branch] gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base 2025-12-04T09:20:54.3104093Z * [new branch] gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head 2025-12-04T09:20:54.3104436Z * [new branch] gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig 2025-12-04T09:20:54.3104764Z * [new branch] gh/alexsamardzic/14/base -> origin/gh/alexsamardzic/14/base 2025-12-04T09:20:54.3109755Z * [new branch] gh/alexsamardzic/14/head -> origin/gh/alexsamardzic/14/head 2025-12-04T09:20:54.3110474Z * [new branch] gh/alexsamardzic/14/orig -> origin/gh/alexsamardzic/14/orig 2025-12-04T09:20:54.3110950Z * [new branch] gh/alexsamardzic/15/base -> origin/gh/alexsamardzic/15/base 2025-12-04T09:20:54.3111289Z * [new branch] gh/alexsamardzic/15/head -> origin/gh/alexsamardzic/15/head 2025-12-04T09:20:54.3111689Z * [new branch] gh/alexsamardzic/15/orig -> origin/gh/alexsamardzic/15/orig 2025-12-04T09:20:54.3112076Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-12-04T09:20:54.3112391Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-12-04T09:20:54.3112700Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-12-04T09:20:54.3113019Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-12-04T09:20:54.3113353Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-12-04T09:20:54.3113675Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-12-04T09:20:54.3113974Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-12-04T09:20:54.3114284Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-12-04T09:20:54.3114597Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-12-04T09:20:54.3118674Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-12-04T09:20:54.3119282Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-12-04T09:20:54.3119775Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-12-04T09:20:54.3120117Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-12-04T09:20:54.3120443Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-12-04T09:20:54.3120816Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-12-04T09:20:54.3123355Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-12-04T09:20:54.3123918Z * [new branch] gh/andyanwang/42/base -> origin/gh/andyanwang/42/base 2025-12-04T09:20:54.3124428Z * [new branch] gh/andyanwang/42/head -> origin/gh/andyanwang/42/head 2025-12-04T09:20:54.3125267Z * [new branch] gh/andyanwang/42/orig -> origin/gh/andyanwang/42/orig 2025-12-04T09:20:54.3125693Z * [new branch] gh/andyanwang/45/base -> origin/gh/andyanwang/45/base 2025-12-04T09:20:54.3126082Z * [new branch] gh/andyanwang/45/head -> origin/gh/andyanwang/45/head 2025-12-04T09:20:54.3126430Z * [new branch] gh/andyanwang/45/orig -> origin/gh/andyanwang/45/orig 2025-12-04T09:20:54.3126782Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-12-04T09:20:54.3127108Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-12-04T09:20:54.3127423Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-12-04T09:20:54.3127761Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-12-04T09:20:54.3128081Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-12-04T09:20:54.3128386Z * [new branch] gh/angelayi/116/base -> origin/gh/angelayi/116/base 2025-12-04T09:20:54.3132045Z * [new branch] gh/angelayi/116/head -> origin/gh/angelayi/116/head 2025-12-04T09:20:54.3132498Z * [new branch] gh/angelayi/116/orig -> origin/gh/angelayi/116/orig 2025-12-04T09:20:54.3132860Z * [new branch] gh/angelayi/122/base -> origin/gh/angelayi/122/base 2025-12-04T09:20:54.3133373Z * [new branch] gh/angelayi/122/head -> origin/gh/angelayi/122/head 2025-12-04T09:20:54.3133713Z * [new branch] gh/angelayi/122/orig -> origin/gh/angelayi/122/orig 2025-12-04T09:20:54.3134060Z * [new branch] gh/angelayi/124/base -> origin/gh/angelayi/124/base 2025-12-04T09:20:54.3134440Z * [new branch] gh/angelayi/124/head -> origin/gh/angelayi/124/head 2025-12-04T09:20:54.3134770Z * [new branch] gh/angelayi/124/orig -> origin/gh/angelayi/124/orig 2025-12-04T09:20:54.3140734Z * [new branch] gh/angelayi/128/base -> origin/gh/angelayi/128/base 2025-12-04T09:20:54.3141143Z * [new branch] gh/angelayi/128/head -> origin/gh/angelayi/128/head 2025-12-04T09:20:54.3141481Z * [new branch] gh/angelayi/128/orig -> origin/gh/angelayi/128/orig 2025-12-04T09:20:54.3141807Z * [new branch] gh/angelayi/131/base -> origin/gh/angelayi/131/base 2025-12-04T09:20:54.3142162Z * [new branch] gh/angelayi/131/head -> origin/gh/angelayi/131/head 2025-12-04T09:20:54.3142495Z * [new branch] gh/angelayi/131/orig -> origin/gh/angelayi/131/orig 2025-12-04T09:20:54.3142834Z * [new branch] gh/angelayi/132/base -> origin/gh/angelayi/132/base 2025-12-04T09:20:54.3143169Z * [new branch] gh/angelayi/132/head -> origin/gh/angelayi/132/head 2025-12-04T09:20:54.3143493Z * [new branch] gh/angelayi/132/orig -> origin/gh/angelayi/132/orig 2025-12-04T09:20:54.3143815Z * [new branch] gh/angelayi/133/base -> origin/gh/angelayi/133/base 2025-12-04T09:20:54.3144138Z * [new branch] gh/angelayi/133/head -> origin/gh/angelayi/133/head 2025-12-04T09:20:54.3144449Z * [new branch] gh/angelayi/133/orig -> origin/gh/angelayi/133/orig 2025-12-04T09:20:54.3144771Z * [new branch] gh/angelayi/134/base -> origin/gh/angelayi/134/base 2025-12-04T09:20:54.3145099Z * [new branch] gh/angelayi/134/head -> origin/gh/angelayi/134/head 2025-12-04T09:20:54.3145427Z * [new branch] gh/angelayi/134/orig -> origin/gh/angelayi/134/orig 2025-12-04T09:20:54.3149919Z * [new branch] gh/angelayi/135/base -> origin/gh/angelayi/135/base 2025-12-04T09:20:54.3150338Z * [new branch] gh/angelayi/135/head -> origin/gh/angelayi/135/head 2025-12-04T09:20:54.3150674Z * [new branch] gh/angelayi/135/orig -> origin/gh/angelayi/135/orig 2025-12-04T09:20:54.3150999Z * [new branch] gh/angelayi/136/base -> origin/gh/angelayi/136/base 2025-12-04T09:20:54.3151313Z * [new branch] gh/angelayi/136/head -> origin/gh/angelayi/136/head 2025-12-04T09:20:54.3151637Z * [new branch] gh/angelayi/136/orig -> origin/gh/angelayi/136/orig 2025-12-04T09:20:54.3152004Z * [new branch] gh/angelayi/137/base -> origin/gh/angelayi/137/base 2025-12-04T09:20:54.3152340Z * [new branch] gh/angelayi/137/head -> origin/gh/angelayi/137/head 2025-12-04T09:20:54.3152659Z * [new branch] gh/angelayi/137/orig -> origin/gh/angelayi/137/orig 2025-12-04T09:20:54.3152985Z * [new branch] gh/angelayi/138/base -> origin/gh/angelayi/138/base 2025-12-04T09:20:54.3153305Z * [new branch] gh/angelayi/138/head -> origin/gh/angelayi/138/head 2025-12-04T09:20:54.3153617Z * [new branch] gh/angelayi/138/orig -> origin/gh/angelayi/138/orig 2025-12-04T09:20:54.3153945Z * [new branch] gh/angelayi/139/base -> origin/gh/angelayi/139/base 2025-12-04T09:20:54.3154253Z * [new branch] gh/angelayi/139/head -> origin/gh/angelayi/139/head 2025-12-04T09:20:54.3154559Z * [new branch] gh/angelayi/139/orig -> origin/gh/angelayi/139/orig 2025-12-04T09:20:54.3160170Z * [new branch] gh/angelayi/140/base -> origin/gh/angelayi/140/base 2025-12-04T09:20:54.3160613Z * [new branch] gh/angelayi/140/head -> origin/gh/angelayi/140/head 2025-12-04T09:20:54.3160966Z * [new branch] gh/angelayi/140/orig -> origin/gh/angelayi/140/orig 2025-12-04T09:20:54.3161451Z * [new branch] gh/angelayi/141/base -> origin/gh/angelayi/141/base 2025-12-04T09:20:54.3161779Z * [new branch] gh/angelayi/141/head -> origin/gh/angelayi/141/head 2025-12-04T09:20:54.3162104Z * [new branch] gh/angelayi/141/orig -> origin/gh/angelayi/141/orig 2025-12-04T09:20:54.3162438Z * [new branch] gh/angelayi/142/base -> origin/gh/angelayi/142/base 2025-12-04T09:20:54.3162760Z * [new branch] gh/angelayi/142/head -> origin/gh/angelayi/142/head 2025-12-04T09:20:54.3163077Z * [new branch] gh/angelayi/142/orig -> origin/gh/angelayi/142/orig 2025-12-04T09:20:54.3163405Z * [new branch] gh/angelayi/143/base -> origin/gh/angelayi/143/base 2025-12-04T09:20:54.3163725Z * [new branch] gh/angelayi/143/head -> origin/gh/angelayi/143/head 2025-12-04T09:20:54.3164044Z * [new branch] gh/angelayi/143/orig -> origin/gh/angelayi/143/orig 2025-12-04T09:20:54.3166866Z * [new branch] gh/angelayi/144/base -> origin/gh/angelayi/144/base 2025-12-04T09:20:54.3167203Z * [new branch] gh/angelayi/144/head -> origin/gh/angelayi/144/head 2025-12-04T09:20:54.3167621Z * [new branch] gh/angelayi/144/orig -> origin/gh/angelayi/144/orig 2025-12-04T09:20:54.3167983Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-12-04T09:20:54.3168354Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-12-04T09:20:54.3168700Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-12-04T09:20:54.3171302Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-12-04T09:20:54.3171775Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-12-04T09:20:54.3172142Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-12-04T09:20:54.3172673Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-12-04T09:20:54.3173030Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-12-04T09:20:54.3173379Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-12-04T09:20:54.3178451Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-12-04T09:20:54.3178907Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-12-04T09:20:54.3179306Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-12-04T09:20:54.3179697Z * [new branch] gh/anijain2305/870/base -> origin/gh/anijain2305/870/base 2025-12-04T09:20:54.3180066Z * [new branch] gh/anijain2305/870/head -> origin/gh/anijain2305/870/head 2025-12-04T09:20:54.3180445Z * [new branch] gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig 2025-12-04T09:20:54.3180808Z * [new branch] gh/anijain2305/873/base -> origin/gh/anijain2305/873/base 2025-12-04T09:20:54.3181408Z * [new branch] gh/anijain2305/873/head -> origin/gh/anijain2305/873/head 2025-12-04T09:20:54.3181818Z * [new branch] gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig 2025-12-04T09:20:54.3182353Z * [new branch] gh/anijain2305/894/base -> origin/gh/anijain2305/894/base 2025-12-04T09:20:54.3182857Z * [new branch] gh/anijain2305/894/head -> origin/gh/anijain2305/894/head 2025-12-04T09:20:54.3183573Z * [new branch] gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig 2025-12-04T09:20:54.3184355Z * [new branch] gh/anijain2305/895/base -> origin/gh/anijain2305/895/base 2025-12-04T09:20:54.3184807Z * [new branch] gh/anijain2305/895/head -> origin/gh/anijain2305/895/head 2025-12-04T09:20:54.3187026Z * [new branch] gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig 2025-12-04T09:20:54.3187717Z * [new branch] gh/anijain2305/910/base -> origin/gh/anijain2305/910/base 2025-12-04T09:20:54.3188262Z * [new branch] gh/anijain2305/910/head -> origin/gh/anijain2305/910/head 2025-12-04T09:20:54.3188833Z * [new branch] gh/anijain2305/910/orig -> origin/gh/anijain2305/910/orig 2025-12-04T09:20:54.3189362Z * [new branch] gh/anijain2305/919/base -> origin/gh/anijain2305/919/base 2025-12-04T09:20:54.3189759Z * [new branch] gh/anijain2305/919/head -> origin/gh/anijain2305/919/head 2025-12-04T09:20:54.3190147Z * [new branch] gh/anijain2305/919/orig -> origin/gh/anijain2305/919/orig 2025-12-04T09:20:54.3190528Z * [new branch] gh/anijain2305/922/base -> origin/gh/anijain2305/922/base 2025-12-04T09:20:54.3190914Z * [new branch] gh/anijain2305/922/head -> origin/gh/anijain2305/922/head 2025-12-04T09:20:54.3191301Z * [new branch] gh/anijain2305/922/orig -> origin/gh/anijain2305/922/orig 2025-12-04T09:20:54.3191667Z * [new branch] gh/anijain2305/932/base -> origin/gh/anijain2305/932/base 2025-12-04T09:20:54.3192190Z * [new branch] gh/anijain2305/932/head -> origin/gh/anijain2305/932/head 2025-12-04T09:20:54.3192961Z * [new branch] gh/anijain2305/932/orig -> origin/gh/anijain2305/932/orig 2025-12-04T09:20:54.3193360Z * [new branch] gh/anijain2305/940/base -> origin/gh/anijain2305/940/base 2025-12-04T09:20:54.3198241Z * [new branch] gh/anijain2305/940/head -> origin/gh/anijain2305/940/head 2025-12-04T09:20:54.3198809Z * [new branch] gh/anijain2305/940/orig -> origin/gh/anijain2305/940/orig 2025-12-04T09:20:54.3199293Z * [new branch] gh/anijain2305/941/base -> origin/gh/anijain2305/941/base 2025-12-04T09:20:54.3200041Z * [new branch] gh/anijain2305/941/head -> origin/gh/anijain2305/941/head 2025-12-04T09:20:54.3200454Z * [new branch] gh/anijain2305/941/orig -> origin/gh/anijain2305/941/orig 2025-12-04T09:20:54.3200799Z * [new branch] gh/anijain2305/942/base -> origin/gh/anijain2305/942/base 2025-12-04T09:20:54.3201144Z * [new branch] gh/anijain2305/942/head -> origin/gh/anijain2305/942/head 2025-12-04T09:20:54.3201485Z * [new branch] gh/anijain2305/942/orig -> origin/gh/anijain2305/942/orig 2025-12-04T09:20:54.3201840Z * [new branch] gh/anijain2305/943/base -> origin/gh/anijain2305/943/base 2025-12-04T09:20:54.3202170Z * [new branch] gh/anijain2305/943/head -> origin/gh/anijain2305/943/head 2025-12-04T09:20:54.3202509Z * [new branch] gh/anijain2305/943/orig -> origin/gh/anijain2305/943/orig 2025-12-04T09:20:54.3202846Z * [new branch] gh/anijain2305/944/base -> origin/gh/anijain2305/944/base 2025-12-04T09:20:54.3203182Z * [new branch] gh/anijain2305/944/head -> origin/gh/anijain2305/944/head 2025-12-04T09:20:54.3203516Z * [new branch] gh/anijain2305/944/orig -> origin/gh/anijain2305/944/orig 2025-12-04T09:20:54.3205923Z * [new branch] gh/anijain2305/945/base -> origin/gh/anijain2305/945/base 2025-12-04T09:20:54.3206276Z * [new branch] gh/anijain2305/945/head -> origin/gh/anijain2305/945/head 2025-12-04T09:20:54.3206610Z * [new branch] gh/anijain2305/945/orig -> origin/gh/anijain2305/945/orig 2025-12-04T09:20:54.3207135Z * [new branch] gh/anijain2305/946/base -> origin/gh/anijain2305/946/base 2025-12-04T09:20:54.3207500Z * [new branch] gh/anijain2305/946/head -> origin/gh/anijain2305/946/head 2025-12-04T09:20:54.3207869Z * [new branch] gh/anijain2305/946/orig -> origin/gh/anijain2305/946/orig 2025-12-04T09:20:54.3208276Z * [new branch] gh/anijain2305/947/base -> origin/gh/anijain2305/947/base 2025-12-04T09:20:54.3208626Z * [new branch] gh/anijain2305/947/head -> origin/gh/anijain2305/947/head 2025-12-04T09:20:54.3208986Z * [new branch] gh/anijain2305/947/orig -> origin/gh/anijain2305/947/orig 2025-12-04T09:20:54.3209380Z * [new branch] gh/anijain2305/948/base -> origin/gh/anijain2305/948/base 2025-12-04T09:20:54.3210041Z * [new branch] gh/anijain2305/948/head -> origin/gh/anijain2305/948/head 2025-12-04T09:20:54.3210713Z * [new branch] gh/anijain2305/948/orig -> origin/gh/anijain2305/948/orig 2025-12-04T09:20:54.3215342Z * [new branch] gh/anijain2305/949/base -> origin/gh/anijain2305/949/base 2025-12-04T09:20:54.3215760Z * [new branch] gh/anijain2305/949/head -> origin/gh/anijain2305/949/head 2025-12-04T09:20:54.3216120Z * [new branch] gh/anijain2305/949/orig -> origin/gh/anijain2305/949/orig 2025-12-04T09:20:54.3216456Z * [new branch] gh/anijain2305/950/base -> origin/gh/anijain2305/950/base 2025-12-04T09:20:54.3216792Z * [new branch] gh/anijain2305/950/head -> origin/gh/anijain2305/950/head 2025-12-04T09:20:54.3217673Z * [new branch] gh/anijain2305/950/orig -> origin/gh/anijain2305/950/orig 2025-12-04T09:20:54.3218109Z * [new branch] gh/anijain2305/951/base -> origin/gh/anijain2305/951/base 2025-12-04T09:20:54.3218490Z * [new branch] gh/anijain2305/951/head -> origin/gh/anijain2305/951/head 2025-12-04T09:20:54.3218876Z * [new branch] gh/anijain2305/951/orig -> origin/gh/anijain2305/951/orig 2025-12-04T09:20:54.3219283Z * [new branch] gh/anijain2305/952/base -> origin/gh/anijain2305/952/base 2025-12-04T09:20:54.3219978Z * [new branch] gh/anijain2305/952/head -> origin/gh/anijain2305/952/head 2025-12-04T09:20:54.3220639Z * [new branch] gh/anijain2305/952/orig -> origin/gh/anijain2305/952/orig 2025-12-04T09:20:54.3223965Z * [new branch] gh/anijain2305/953/base -> origin/gh/anijain2305/953/base 2025-12-04T09:20:54.3224563Z * [new branch] gh/anijain2305/953/head -> origin/gh/anijain2305/953/head 2025-12-04T09:20:54.3225067Z * [new branch] gh/anijain2305/953/orig -> origin/gh/anijain2305/953/orig 2025-12-04T09:20:54.3225459Z * [new branch] gh/anijain2305/954/base -> origin/gh/anijain2305/954/base 2025-12-04T09:20:54.3225847Z * [new branch] gh/anijain2305/954/head -> origin/gh/anijain2305/954/head 2025-12-04T09:20:54.3226216Z * [new branch] gh/anijain2305/954/orig -> origin/gh/anijain2305/954/orig 2025-12-04T09:20:54.3226739Z * [new branch] gh/anijain2305/955/base -> origin/gh/anijain2305/955/base 2025-12-04T09:20:54.3227400Z * [new branch] gh/anijain2305/955/head -> origin/gh/anijain2305/955/head 2025-12-04T09:20:54.3228144Z * [new branch] gh/anijain2305/955/orig -> origin/gh/anijain2305/955/orig 2025-12-04T09:20:54.3232938Z * [new branch] gh/anijain2305/956/base -> origin/gh/anijain2305/956/base 2025-12-04T09:20:54.3233354Z * [new branch] gh/anijain2305/956/head -> origin/gh/anijain2305/956/head 2025-12-04T09:20:54.3233703Z * [new branch] gh/anijain2305/956/orig -> origin/gh/anijain2305/956/orig 2025-12-04T09:20:54.3234031Z * [new branch] gh/anijain2305/957/base -> origin/gh/anijain2305/957/base 2025-12-04T09:20:54.3234530Z * [new branch] gh/anijain2305/957/head -> origin/gh/anijain2305/957/head 2025-12-04T09:20:54.3234869Z * [new branch] gh/anijain2305/957/orig -> origin/gh/anijain2305/957/orig 2025-12-04T09:20:54.3235372Z * [new branch] gh/anijain2305/958/base -> origin/gh/anijain2305/958/base 2025-12-04T09:20:54.3236121Z * [new branch] gh/anijain2305/958/head -> origin/gh/anijain2305/958/head 2025-12-04T09:20:54.3236600Z * [new branch] gh/anijain2305/958/orig -> origin/gh/anijain2305/958/orig 2025-12-04T09:20:54.3237087Z * [new branch] gh/anijain2305/959/base -> origin/gh/anijain2305/959/base 2025-12-04T09:20:54.3237554Z * [new branch] gh/anijain2305/959/head -> origin/gh/anijain2305/959/head 2025-12-04T09:20:54.3238018Z * [new branch] gh/anijain2305/959/orig -> origin/gh/anijain2305/959/orig 2025-12-04T09:20:54.3240719Z * [new branch] gh/anijain2305/960/base -> origin/gh/anijain2305/960/base 2025-12-04T09:20:54.3241394Z * [new branch] gh/anijain2305/960/head -> origin/gh/anijain2305/960/head 2025-12-04T09:20:54.3241867Z * [new branch] gh/anijain2305/960/orig -> origin/gh/anijain2305/960/orig 2025-12-04T09:20:54.3242226Z * [new branch] gh/anijain2305/961/base -> origin/gh/anijain2305/961/base 2025-12-04T09:20:54.3242570Z * [new branch] gh/anijain2305/961/head -> origin/gh/anijain2305/961/head 2025-12-04T09:20:54.3242907Z * [new branch] gh/anijain2305/961/orig -> origin/gh/anijain2305/961/orig 2025-12-04T09:20:54.3243423Z * [new branch] gh/anijain2305/962/base -> origin/gh/anijain2305/962/base 2025-12-04T09:20:54.3243981Z * [new branch] gh/anijain2305/962/head -> origin/gh/anijain2305/962/head 2025-12-04T09:20:54.3244632Z * [new branch] gh/anijain2305/962/orig -> origin/gh/anijain2305/962/orig 2025-12-04T09:20:54.3249777Z * [new branch] gh/anijain2305/963/base -> origin/gh/anijain2305/963/base 2025-12-04T09:20:54.3250202Z * [new branch] gh/anijain2305/963/head -> origin/gh/anijain2305/963/head 2025-12-04T09:20:54.3250554Z * [new branch] gh/anijain2305/963/orig -> origin/gh/anijain2305/963/orig 2025-12-04T09:20:54.3250906Z * [new branch] gh/anijain2305/964/base -> origin/gh/anijain2305/964/base 2025-12-04T09:20:54.3251245Z * [new branch] gh/anijain2305/964/head -> origin/gh/anijain2305/964/head 2025-12-04T09:20:54.3251782Z * [new branch] gh/anijain2305/964/orig -> origin/gh/anijain2305/964/orig 2025-12-04T09:20:54.3252155Z * [new branch] gh/anijain2305/965/base -> origin/gh/anijain2305/965/base 2025-12-04T09:20:54.3252511Z * [new branch] gh/anijain2305/965/head -> origin/gh/anijain2305/965/head 2025-12-04T09:20:54.3252878Z * [new branch] gh/anijain2305/965/orig -> origin/gh/anijain2305/965/orig 2025-12-04T09:20:54.3253429Z * [new branch] gh/anijain2305/966/base -> origin/gh/anijain2305/966/base 2025-12-04T09:20:54.3253854Z * [new branch] gh/anijain2305/966/head -> origin/gh/anijain2305/966/head 2025-12-04T09:20:54.3254367Z * [new branch] gh/anijain2305/966/orig -> origin/gh/anijain2305/966/orig 2025-12-04T09:20:54.3255998Z * [new branch] gh/anijain2305/967/base -> origin/gh/anijain2305/967/base 2025-12-04T09:20:54.3256547Z * [new branch] gh/anijain2305/967/head -> origin/gh/anijain2305/967/head 2025-12-04T09:20:54.3256916Z * [new branch] gh/anijain2305/967/orig -> origin/gh/anijain2305/967/orig 2025-12-04T09:20:54.3258888Z * [new branch] gh/anijain2305/968/base -> origin/gh/anijain2305/968/base 2025-12-04T09:20:54.3259295Z * [new branch] gh/anijain2305/968/head -> origin/gh/anijain2305/968/head 2025-12-04T09:20:54.3259793Z * [new branch] gh/anijain2305/968/orig -> origin/gh/anijain2305/968/orig 2025-12-04T09:20:54.3260185Z * [new branch] gh/anijain2305/969/base -> origin/gh/anijain2305/969/base 2025-12-04T09:20:54.3260954Z * [new branch] gh/anijain2305/969/head -> origin/gh/anijain2305/969/head 2025-12-04T09:20:54.3261669Z * [new branch] gh/anijain2305/969/orig -> origin/gh/anijain2305/969/orig 2025-12-04T09:20:54.3262876Z * [new branch] gh/anijain2305/970/base -> origin/gh/anijain2305/970/base 2025-12-04T09:20:54.3263213Z * [new branch] gh/anijain2305/970/head -> origin/gh/anijain2305/970/head 2025-12-04T09:20:54.3263922Z * [new branch] gh/anijain2305/970/orig -> origin/gh/anijain2305/970/orig 2025-12-04T09:20:54.3268366Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-12-04T09:20:54.3268751Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-12-04T09:20:54.3269095Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-12-04T09:20:54.3269425Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-12-04T09:20:54.3269767Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-12-04T09:20:54.3274670Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-12-04T09:20:54.3276318Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-12-04T09:20:54.3276644Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-12-04T09:20:54.3276944Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-12-04T09:20:54.3277245Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-12-04T09:20:54.3277548Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-12-04T09:20:54.3277836Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-12-04T09:20:54.3278124Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-12-04T09:20:54.3278434Z * [new branch] gh/anshul-si/53/base -> origin/gh/anshul-si/53/base 2025-12-04T09:20:54.3278742Z * [new branch] gh/anshul-si/53/head -> origin/gh/anshul-si/53/head 2025-12-04T09:20:54.3279034Z * [new branch] gh/anshul-si/58/base -> origin/gh/anshul-si/58/base 2025-12-04T09:20:54.3279531Z * [new branch] gh/anshul-si/58/head -> origin/gh/anshul-si/58/head 2025-12-04T09:20:54.3279835Z * [new branch] gh/anshul-si/66/base -> origin/gh/anshul-si/66/base 2025-12-04T09:20:54.3280126Z * [new branch] gh/anshul-si/66/head -> origin/gh/anshul-si/66/head 2025-12-04T09:20:54.3280515Z * [new branch] gh/anshul-si/66/orig -> origin/gh/anshul-si/66/orig 2025-12-04T09:20:54.3284923Z * [new branch] gh/anshul-si/67/base -> origin/gh/anshul-si/67/base 2025-12-04T09:20:54.3289643Z * [new branch] gh/anshul-si/67/head -> origin/gh/anshul-si/67/head 2025-12-04T09:20:54.3294015Z * [new branch] gh/anshul-si/67/orig -> origin/gh/anshul-si/67/orig 2025-12-04T09:20:54.3298754Z * [new branch] gh/anshul-si/68/base -> origin/gh/anshul-si/68/base 2025-12-04T09:20:54.3304202Z * [new branch] gh/anshul-si/68/head -> origin/gh/anshul-si/68/head 2025-12-04T09:20:54.3306450Z * [new branch] gh/anshul-si/68/orig -> origin/gh/anshul-si/68/orig 2025-12-04T09:20:54.3306919Z * [new branch] gh/anshul-si/69/base -> origin/gh/anshul-si/69/base 2025-12-04T09:20:54.3313033Z * [new branch] gh/anshul-si/69/head -> origin/gh/anshul-si/69/head 2025-12-04T09:20:54.3314767Z * [new branch] gh/anshul-si/69/orig -> origin/gh/anshul-si/69/orig 2025-12-04T09:20:54.3315144Z * [new branch] gh/anshul-si/70/base -> origin/gh/anshul-si/70/base 2025-12-04T09:20:54.3315473Z * [new branch] gh/anshul-si/70/head -> origin/gh/anshul-si/70/head 2025-12-04T09:20:54.3315877Z * [new branch] gh/anshul-si/70/orig -> origin/gh/anshul-si/70/orig 2025-12-04T09:20:54.3316201Z * [new branch] gh/anshul-si/71/base -> origin/gh/anshul-si/71/base 2025-12-04T09:20:54.3316510Z * [new branch] gh/anshul-si/71/head -> origin/gh/anshul-si/71/head 2025-12-04T09:20:54.3316818Z * [new branch] gh/anshul-si/71/orig -> origin/gh/anshul-si/71/orig 2025-12-04T09:20:54.3317118Z * [new branch] gh/anshul-si/72/base -> origin/gh/anshul-si/72/base 2025-12-04T09:20:54.3317424Z * [new branch] gh/anshul-si/72/head -> origin/gh/anshul-si/72/head 2025-12-04T09:20:54.3317739Z * [new branch] gh/anshul-si/72/orig -> origin/gh/anshul-si/72/orig 2025-12-04T09:20:54.3318044Z * [new branch] gh/anshul-si/73/base -> origin/gh/anshul-si/73/base 2025-12-04T09:20:54.3318344Z * [new branch] gh/anshul-si/73/head -> origin/gh/anshul-si/73/head 2025-12-04T09:20:54.3318660Z * [new branch] gh/anshul-si/73/orig -> origin/gh/anshul-si/73/orig 2025-12-04T09:20:54.3318966Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-12-04T09:20:54.3319273Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-12-04T09:20:54.3319576Z * [new branch] gh/aorenste/134/base -> origin/gh/aorenste/134/base 2025-12-04T09:20:54.3319894Z * [new branch] gh/aorenste/134/head -> origin/gh/aorenste/134/head 2025-12-04T09:20:54.3320215Z * [new branch] gh/aorenste/134/orig -> origin/gh/aorenste/134/orig 2025-12-04T09:20:54.3320526Z * [new branch] gh/aorenste/139/base -> origin/gh/aorenste/139/base 2025-12-04T09:20:54.3320847Z * [new branch] gh/aorenste/139/head -> origin/gh/aorenste/139/head 2025-12-04T09:20:54.3321168Z * [new branch] gh/aorenste/139/orig -> origin/gh/aorenste/139/orig 2025-12-04T09:20:54.3321483Z * [new branch] gh/aorenste/141/base -> origin/gh/aorenste/141/base 2025-12-04T09:20:54.3321796Z * [new branch] gh/aorenste/141/head -> origin/gh/aorenste/141/head 2025-12-04T09:20:54.3322129Z * [new branch] gh/aorenste/145/base -> origin/gh/aorenste/145/base 2025-12-04T09:20:54.3322465Z * [new branch] gh/aorenste/145/head -> origin/gh/aorenste/145/head 2025-12-04T09:20:54.3322770Z * [new branch] gh/aorenste/145/orig -> origin/gh/aorenste/145/orig 2025-12-04T09:20:54.3323125Z * [new branch] gh/aorenste/146/base -> origin/gh/aorenste/146/base 2025-12-04T09:20:54.3323438Z * [new branch] gh/aorenste/146/head -> origin/gh/aorenste/146/head 2025-12-04T09:20:54.3323752Z * [new branch] gh/aorenste/146/orig -> origin/gh/aorenste/146/orig 2025-12-04T09:20:54.3324067Z * [new branch] gh/aorenste/147/base -> origin/gh/aorenste/147/base 2025-12-04T09:20:54.3324539Z * [new branch] gh/aorenste/147/head -> origin/gh/aorenste/147/head 2025-12-04T09:20:54.3324854Z * [new branch] gh/aorenste/147/orig -> origin/gh/aorenste/147/orig 2025-12-04T09:20:54.3325168Z * [new branch] gh/aorenste/148/base -> origin/gh/aorenste/148/base 2025-12-04T09:20:54.3325478Z * [new branch] gh/aorenste/148/head -> origin/gh/aorenste/148/head 2025-12-04T09:20:54.3325793Z * [new branch] gh/aorenste/148/orig -> origin/gh/aorenste/148/orig 2025-12-04T09:20:54.3326167Z * [new branch] gh/aorenste/149/base -> origin/gh/aorenste/149/base 2025-12-04T09:20:54.3326503Z * [new branch] gh/aorenste/149/head -> origin/gh/aorenste/149/head 2025-12-04T09:20:54.3326843Z * [new branch] gh/aorenste/149/orig -> origin/gh/aorenste/149/orig 2025-12-04T09:20:54.3327299Z * [new branch] gh/aorenste/150/base -> origin/gh/aorenste/150/base 2025-12-04T09:20:54.3327636Z * [new branch] gh/aorenste/150/head -> origin/gh/aorenste/150/head 2025-12-04T09:20:54.3327972Z * [new branch] gh/aorenste/150/orig -> origin/gh/aorenste/150/orig 2025-12-04T09:20:54.3328301Z * [new branch] gh/aorenste/151/base -> origin/gh/aorenste/151/base 2025-12-04T09:20:54.3328637Z * [new branch] gh/aorenste/151/head -> origin/gh/aorenste/151/head 2025-12-04T09:20:54.3328977Z * [new branch] gh/aorenste/151/orig -> origin/gh/aorenste/151/orig 2025-12-04T09:20:54.3329317Z * [new branch] gh/aorenste/152/base -> origin/gh/aorenste/152/base 2025-12-04T09:20:54.3329650Z * [new branch] gh/aorenste/152/head -> origin/gh/aorenste/152/head 2025-12-04T09:20:54.3329985Z * [new branch] gh/aorenste/152/orig -> origin/gh/aorenste/152/orig 2025-12-04T09:20:54.3330326Z * [new branch] gh/aorenste/153/base -> origin/gh/aorenste/153/base 2025-12-04T09:20:54.3330653Z * [new branch] gh/aorenste/153/head -> origin/gh/aorenste/153/head 2025-12-04T09:20:54.3330995Z * [new branch] gh/aorenste/153/orig -> origin/gh/aorenste/153/orig 2025-12-04T09:20:54.3331344Z * [new branch] gh/aorenste/154/base -> origin/gh/aorenste/154/base 2025-12-04T09:20:54.3331991Z * [new branch] gh/aorenste/154/head -> origin/gh/aorenste/154/head 2025-12-04T09:20:54.3332591Z * [new branch] gh/aorenste/154/orig -> origin/gh/aorenste/154/orig 2025-12-04T09:20:54.3333181Z * [new branch] gh/aorenste/155/base -> origin/gh/aorenste/155/base 2025-12-04T09:20:54.3333639Z * [new branch] gh/aorenste/155/head -> origin/gh/aorenste/155/head 2025-12-04T09:20:54.3333966Z * [new branch] gh/aorenste/155/orig -> origin/gh/aorenste/155/orig 2025-12-04T09:20:54.3334278Z * [new branch] gh/aorenste/156/base -> origin/gh/aorenste/156/base 2025-12-04T09:20:54.3334598Z * [new branch] gh/aorenste/156/head -> origin/gh/aorenste/156/head 2025-12-04T09:20:54.3334919Z * [new branch] gh/aorenste/156/orig -> origin/gh/aorenste/156/orig 2025-12-04T09:20:54.3335245Z * [new branch] gh/aorenste/157/base -> origin/gh/aorenste/157/base 2025-12-04T09:20:54.3335573Z * [new branch] gh/aorenste/157/head -> origin/gh/aorenste/157/head 2025-12-04T09:20:54.3335914Z * [new branch] gh/aorenste/157/orig -> origin/gh/aorenste/157/orig 2025-12-04T09:20:54.3336252Z * [new branch] gh/aorenste/158/base -> origin/gh/aorenste/158/base 2025-12-04T09:20:54.3336589Z * [new branch] gh/aorenste/158/head -> origin/gh/aorenste/158/head 2025-12-04T09:20:54.3336928Z * [new branch] gh/aorenste/158/orig -> origin/gh/aorenste/158/orig 2025-12-04T09:20:54.3337721Z * [new branch] gh/aorenste/159/base -> origin/gh/aorenste/159/base 2025-12-04T09:20:54.3338136Z * [new branch] gh/aorenste/159/head -> origin/gh/aorenste/159/head 2025-12-04T09:20:54.3338482Z * [new branch] gh/aorenste/159/orig -> origin/gh/aorenste/159/orig 2025-12-04T09:20:54.3339834Z * [new branch] gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base 2025-12-04T09:20:54.3343051Z * [new branch] gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head 2025-12-04T09:20:54.3343534Z * [new branch] gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base 2025-12-04T09:20:54.3343875Z * [new branch] gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head 2025-12-04T09:20:54.3344199Z * [new branch] gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig 2025-12-04T09:20:54.3344616Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-12-04T09:20:54.3347549Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-12-04T09:20:54.3347864Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-12-04T09:20:54.3348163Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-12-04T09:20:54.3348464Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-12-04T09:20:54.3348769Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-12-04T09:20:54.3349086Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-12-04T09:20:54.3352187Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-12-04T09:20:54.3352498Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-12-04T09:20:54.3352812Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-12-04T09:20:54.3353116Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-12-04T09:20:54.3353413Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-12-04T09:20:54.3353719Z * [new branch] gh/bdhirsh/672/base -> origin/gh/bdhirsh/672/base 2025-12-04T09:20:54.3355766Z * [new branch] gh/bdhirsh/672/head -> origin/gh/bdhirsh/672/head 2025-12-04T09:20:54.3356078Z * [new branch] gh/bdhirsh/672/orig -> origin/gh/bdhirsh/672/orig 2025-12-04T09:20:54.3356375Z * [new branch] gh/bdhirsh/675/base -> origin/gh/bdhirsh/675/base 2025-12-04T09:20:54.3356683Z * [new branch] gh/bdhirsh/675/head -> origin/gh/bdhirsh/675/head 2025-12-04T09:20:54.3358956Z * [new branch] gh/bdhirsh/675/orig -> origin/gh/bdhirsh/675/orig 2025-12-04T09:20:54.3359258Z * [new branch] gh/bdhirsh/676/base -> origin/gh/bdhirsh/676/base 2025-12-04T09:20:54.3359547Z * [new branch] gh/bdhirsh/676/head -> origin/gh/bdhirsh/676/head 2025-12-04T09:20:54.3359845Z * [new branch] gh/bdhirsh/676/orig -> origin/gh/bdhirsh/676/orig 2025-12-04T09:20:54.3364163Z * [new branch] gh/bdhirsh/677/base -> origin/gh/bdhirsh/677/base 2025-12-04T09:20:54.3364292Z * [new branch] gh/bdhirsh/677/head -> origin/gh/bdhirsh/677/head 2025-12-04T09:20:54.3364418Z * [new branch] gh/bdhirsh/677/orig -> origin/gh/bdhirsh/677/orig 2025-12-04T09:20:54.3364545Z * [new branch] gh/bdhirsh/678/base -> origin/gh/bdhirsh/678/base 2025-12-04T09:20:54.3364665Z * [new branch] gh/bdhirsh/678/head -> origin/gh/bdhirsh/678/head 2025-12-04T09:20:54.3364800Z * [new branch] gh/bdhirsh/678/orig -> origin/gh/bdhirsh/678/orig 2025-12-04T09:20:54.3365090Z * [new branch] gh/bdhirsh/679/base -> origin/gh/bdhirsh/679/base 2025-12-04T09:20:54.3366598Z * [new branch] gh/bdhirsh/679/head -> origin/gh/bdhirsh/679/head 2025-12-04T09:20:54.3366769Z * [new branch] gh/bdhirsh/679/orig -> origin/gh/bdhirsh/679/orig 2025-12-04T09:20:54.3367541Z * [new branch] gh/bdhirsh/680/base -> origin/gh/bdhirsh/680/base 2025-12-04T09:20:54.3368366Z * [new branch] gh/bdhirsh/680/head -> origin/gh/bdhirsh/680/head 2025-12-04T09:20:54.3368792Z * [new branch] gh/bdhirsh/680/orig -> origin/gh/bdhirsh/680/orig 2025-12-04T09:20:54.3372713Z * [new branch] gh/bdhirsh/681/base -> origin/gh/bdhirsh/681/base 2025-12-04T09:20:54.3372904Z * [new branch] gh/bdhirsh/681/head -> origin/gh/bdhirsh/681/head 2025-12-04T09:20:54.3373251Z * [new branch] gh/bdhirsh/681/orig -> origin/gh/bdhirsh/681/orig 2025-12-04T09:20:54.3373438Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-12-04T09:20:54.3376438Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-12-04T09:20:54.3376783Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-12-04T09:20:54.3376963Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-12-04T09:20:54.3378313Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-12-04T09:20:54.3378657Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-12-04T09:20:54.3384354Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-12-04T09:20:54.3389237Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-12-04T09:20:54.3394449Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-12-04T09:20:54.3400025Z * [new branch] gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base 2025-12-04T09:20:54.3402139Z * [new branch] gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head 2025-12-04T09:20:54.3402316Z * [new branch] gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig 2025-12-04T09:20:54.3402462Z * [new branch] gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base 2025-12-04T09:20:54.3402620Z * [new branch] gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head 2025-12-04T09:20:54.3402771Z * [new branch] gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig 2025-12-04T09:20:54.3402916Z * [new branch] gh/benjaminglass1/109/base -> origin/gh/benjaminglass1/109/base 2025-12-04T09:20:54.3403073Z * [new branch] gh/benjaminglass1/109/head -> origin/gh/benjaminglass1/109/head 2025-12-04T09:20:54.3403226Z * [new branch] gh/benjaminglass1/109/orig -> origin/gh/benjaminglass1/109/orig 2025-12-04T09:20:54.3403410Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-12-04T09:20:54.3403563Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-12-04T09:20:54.3403704Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-12-04T09:20:54.3403847Z * [new branch] gh/bobrenjc93/570/base -> origin/gh/bobrenjc93/570/base 2025-12-04T09:20:54.3403987Z * [new branch] gh/bobrenjc93/570/head -> origin/gh/bobrenjc93/570/head 2025-12-04T09:20:54.3404118Z * [new branch] gh/bobrenjc93/570/orig -> origin/gh/bobrenjc93/570/orig 2025-12-04T09:20:54.3404267Z * [new branch] gh/bobrenjc93/604/base -> origin/gh/bobrenjc93/604/base 2025-12-04T09:20:54.3404396Z * [new branch] gh/bobrenjc93/604/head -> origin/gh/bobrenjc93/604/head 2025-12-04T09:20:54.3404522Z * [new branch] gh/bobrenjc93/604/orig -> origin/gh/bobrenjc93/604/orig 2025-12-04T09:20:54.3404655Z * [new branch] gh/bobrenjc93/638/base -> origin/gh/bobrenjc93/638/base 2025-12-04T09:20:54.3404782Z * [new branch] gh/bobrenjc93/638/head -> origin/gh/bobrenjc93/638/head 2025-12-04T09:20:54.3404913Z * [new branch] gh/bobrenjc93/638/orig -> origin/gh/bobrenjc93/638/orig 2025-12-04T09:20:54.3405263Z * [new branch] gh/bobrenjc93/653/base -> origin/gh/bobrenjc93/653/base 2025-12-04T09:20:54.3405400Z * [new branch] gh/bobrenjc93/653/head -> origin/gh/bobrenjc93/653/head 2025-12-04T09:20:54.3405542Z * [new branch] gh/bobrenjc93/653/orig -> origin/gh/bobrenjc93/653/orig 2025-12-04T09:20:54.3405729Z * [new branch] gh/bobrenjc93/654/base -> origin/gh/bobrenjc93/654/base 2025-12-04T09:20:54.3405881Z * [new branch] gh/bobrenjc93/654/head -> origin/gh/bobrenjc93/654/head 2025-12-04T09:20:54.3406021Z * [new branch] gh/bobrenjc93/654/orig -> origin/gh/bobrenjc93/654/orig 2025-12-04T09:20:54.3406162Z * [new branch] gh/bobrenjc93/657/base -> origin/gh/bobrenjc93/657/base 2025-12-04T09:20:54.3406307Z * [new branch] gh/bobrenjc93/657/head -> origin/gh/bobrenjc93/657/head 2025-12-04T09:20:54.3406444Z * [new branch] gh/bobrenjc93/657/orig -> origin/gh/bobrenjc93/657/orig 2025-12-04T09:20:54.3406595Z * [new branch] gh/bobrenjc93/672/base -> origin/gh/bobrenjc93/672/base 2025-12-04T09:20:54.3407272Z * [new branch] gh/bobrenjc93/672/head -> origin/gh/bobrenjc93/672/head 2025-12-04T09:20:54.3407503Z * [new branch] gh/bobrenjc93/672/orig -> origin/gh/bobrenjc93/672/orig 2025-12-04T09:20:54.3408098Z * [new branch] gh/bobrenjc93/679/base -> origin/gh/bobrenjc93/679/base 2025-12-04T09:20:54.3408394Z * [new branch] gh/bobrenjc93/679/head -> origin/gh/bobrenjc93/679/head 2025-12-04T09:20:54.3408656Z * [new branch] gh/bobrenjc93/679/orig -> origin/gh/bobrenjc93/679/orig 2025-12-04T09:20:54.3409119Z * [new branch] gh/bobrenjc93/680/base -> origin/gh/bobrenjc93/680/base 2025-12-04T09:20:54.3409497Z * [new branch] gh/bobrenjc93/680/head -> origin/gh/bobrenjc93/680/head 2025-12-04T09:20:54.3409661Z * [new branch] gh/bobrenjc93/680/orig -> origin/gh/bobrenjc93/680/orig 2025-12-04T09:20:54.3409837Z * [new branch] gh/bobrenjc93/681/base -> origin/gh/bobrenjc93/681/base 2025-12-04T09:20:54.3416204Z * [new branch] gh/bobrenjc93/681/head -> origin/gh/bobrenjc93/681/head 2025-12-04T09:20:54.3416543Z * [new branch] gh/bobrenjc93/681/orig -> origin/gh/bobrenjc93/681/orig 2025-12-04T09:20:54.3416720Z * [new branch] gh/bobrenjc93/682/base -> origin/gh/bobrenjc93/682/base 2025-12-04T09:20:54.3416857Z * [new branch] gh/bobrenjc93/682/head -> origin/gh/bobrenjc93/682/head 2025-12-04T09:20:54.3417000Z * [new branch] gh/bobrenjc93/682/orig -> origin/gh/bobrenjc93/682/orig 2025-12-04T09:20:54.3417135Z * [new branch] gh/bobrenjc93/683/base -> origin/gh/bobrenjc93/683/base 2025-12-04T09:20:54.3417411Z * [new branch] gh/bobrenjc93/683/head -> origin/gh/bobrenjc93/683/head 2025-12-04T09:20:54.3417583Z * [new branch] gh/bobrenjc93/683/orig -> origin/gh/bobrenjc93/683/orig 2025-12-04T09:20:54.3417723Z * [new branch] gh/bobrenjc93/684/base -> origin/gh/bobrenjc93/684/base 2025-12-04T09:20:54.3417976Z * [new branch] gh/bobrenjc93/684/head -> origin/gh/bobrenjc93/684/head 2025-12-04T09:20:54.3420335Z * [new branch] gh/bobrenjc93/684/orig -> origin/gh/bobrenjc93/684/orig 2025-12-04T09:20:54.3420534Z * [new branch] gh/bobrenjc93/685/base -> origin/gh/bobrenjc93/685/base 2025-12-04T09:20:54.3420689Z * [new branch] gh/bobrenjc93/685/head -> origin/gh/bobrenjc93/685/head 2025-12-04T09:20:54.3422516Z * [new branch] gh/bobrenjc93/685/orig -> origin/gh/bobrenjc93/685/orig 2025-12-04T09:20:54.3422710Z * [new branch] gh/bobrenjc93/686/base -> origin/gh/bobrenjc93/686/base 2025-12-04T09:20:54.3423184Z * [new branch] gh/bobrenjc93/686/head -> origin/gh/bobrenjc93/686/head 2025-12-04T09:20:54.3425734Z * [new branch] gh/bobrenjc93/686/orig -> origin/gh/bobrenjc93/686/orig 2025-12-04T09:20:54.3426095Z * [new branch] gh/bobrenjc93/687/base -> origin/gh/bobrenjc93/687/base 2025-12-04T09:20:54.3426501Z * [new branch] gh/bobrenjc93/687/head -> origin/gh/bobrenjc93/687/head 2025-12-04T09:20:54.3426802Z * [new branch] gh/bobrenjc93/687/orig -> origin/gh/bobrenjc93/687/orig 2025-12-04T09:20:54.3428016Z * [new branch] gh/bobrenjc93/688/base -> origin/gh/bobrenjc93/688/base 2025-12-04T09:20:54.3428189Z * [new branch] gh/bobrenjc93/688/head -> origin/gh/bobrenjc93/688/head 2025-12-04T09:20:54.3431798Z * [new branch] gh/bobrenjc93/688/orig -> origin/gh/bobrenjc93/688/orig 2025-12-04T09:20:54.3436289Z * [new branch] gh/bobrenjc93/689/base -> origin/gh/bobrenjc93/689/base 2025-12-04T09:20:54.3436493Z * [new branch] gh/bobrenjc93/689/head -> origin/gh/bobrenjc93/689/head 2025-12-04T09:20:54.3436650Z * [new branch] gh/bobrenjc93/689/orig -> origin/gh/bobrenjc93/689/orig 2025-12-04T09:20:54.3436793Z * [new branch] gh/bobrenjc93/690/base -> origin/gh/bobrenjc93/690/base 2025-12-04T09:20:54.3436955Z * [new branch] gh/bobrenjc93/690/head -> origin/gh/bobrenjc93/690/head 2025-12-04T09:20:54.3437104Z * [new branch] gh/bobrenjc93/690/orig -> origin/gh/bobrenjc93/690/orig 2025-12-04T09:20:54.3441683Z * [new branch] gh/bobrenjc93/691/base -> origin/gh/bobrenjc93/691/base 2025-12-04T09:20:54.3442028Z * [new branch] gh/bobrenjc93/691/head -> origin/gh/bobrenjc93/691/head 2025-12-04T09:20:54.3442213Z * [new branch] gh/bobrenjc93/691/orig -> origin/gh/bobrenjc93/691/orig 2025-12-04T09:20:54.3442398Z * [new branch] gh/bobrenjc93/692/base -> origin/gh/bobrenjc93/692/base 2025-12-04T09:20:54.3442596Z * [new branch] gh/bobrenjc93/692/head -> origin/gh/bobrenjc93/692/head 2025-12-04T09:20:54.3442752Z * [new branch] gh/bobrenjc93/692/orig -> origin/gh/bobrenjc93/692/orig 2025-12-04T09:20:54.3446932Z * [new branch] gh/bobrenjc93/693/base -> origin/gh/bobrenjc93/693/base 2025-12-04T09:20:54.3447319Z * [new branch] gh/bobrenjc93/693/head -> origin/gh/bobrenjc93/693/head 2025-12-04T09:20:54.3447573Z * [new branch] gh/bobrenjc93/693/orig -> origin/gh/bobrenjc93/693/orig 2025-12-04T09:20:54.3447828Z * [new branch] gh/bobrenjc93/694/base -> origin/gh/bobrenjc93/694/base 2025-12-04T09:20:54.3448003Z * [new branch] gh/bobrenjc93/694/head -> origin/gh/bobrenjc93/694/head 2025-12-04T09:20:54.3448242Z * [new branch] gh/bobrenjc93/694/orig -> origin/gh/bobrenjc93/694/orig 2025-12-04T09:20:54.3448422Z * [new branch] gh/bobrenjc93/695/base -> origin/gh/bobrenjc93/695/base 2025-12-04T09:20:54.3448680Z * [new branch] gh/bobrenjc93/695/head -> origin/gh/bobrenjc93/695/head 2025-12-04T09:20:54.3448831Z * [new branch] gh/bobrenjc93/695/orig -> origin/gh/bobrenjc93/695/orig 2025-12-04T09:20:54.3448976Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-12-04T09:20:54.3449113Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-12-04T09:20:54.3449666Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-12-04T09:20:54.3449826Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-12-04T09:20:54.3449970Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-12-04T09:20:54.3450773Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-12-04T09:20:54.3451411Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-12-04T09:20:54.3452305Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-12-04T09:20:54.3457294Z * [new branch] gh/c00w/56/base -> origin/gh/c00w/56/base 2025-12-04T09:20:54.3457657Z * [new branch] gh/c00w/56/head -> origin/gh/c00w/56/head 2025-12-04T09:20:54.3457797Z * [new branch] gh/c00w/56/orig -> origin/gh/c00w/56/orig 2025-12-04T09:20:54.3457922Z * [new branch] gh/c00w/57/base -> origin/gh/c00w/57/base 2025-12-04T09:20:54.3458071Z * [new branch] gh/c00w/57/head -> origin/gh/c00w/57/head 2025-12-04T09:20:54.3458210Z * [new branch] gh/c00w/57/orig -> origin/gh/c00w/57/orig 2025-12-04T09:20:54.3461321Z * [new branch] gh/c00w/58/base -> origin/gh/c00w/58/base 2025-12-04T09:20:54.3461494Z * [new branch] gh/c00w/58/head -> origin/gh/c00w/58/head 2025-12-04T09:20:54.3461630Z * [new branch] gh/c00w/58/orig -> origin/gh/c00w/58/orig 2025-12-04T09:20:54.3461779Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-12-04T09:20:54.3461942Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-12-04T09:20:54.3462077Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-12-04T09:20:54.3466786Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-12-04T09:20:54.3469710Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-12-04T09:20:54.3469917Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-12-04T09:20:54.3470083Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-12-04T09:20:54.3470249Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-12-04T09:20:54.3470400Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-12-04T09:20:54.3470556Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-12-04T09:20:54.3470715Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-12-04T09:20:54.3476273Z * [new branch] gh/coconutruben/70/base -> origin/gh/coconutruben/70/base 2025-12-04T09:20:54.3479768Z * [new branch] gh/coconutruben/70/head -> origin/gh/coconutruben/70/head 2025-12-04T09:20:54.3483959Z * [new branch] gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig 2025-12-04T09:20:54.3485847Z * [new branch] gh/coconutruben/71/base -> origin/gh/coconutruben/71/base 2025-12-04T09:20:54.3486002Z * [new branch] gh/coconutruben/71/head -> origin/gh/coconutruben/71/head 2025-12-04T09:20:54.3486188Z * [new branch] gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig 2025-12-04T09:20:54.3486338Z * [new branch] gh/coconutruben/72/base -> origin/gh/coconutruben/72/base 2025-12-04T09:20:54.3486494Z * [new branch] gh/coconutruben/72/head -> origin/gh/coconutruben/72/head 2025-12-04T09:20:54.3486650Z * [new branch] gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig 2025-12-04T09:20:54.3486794Z * [new branch] gh/coconutruben/73/base -> origin/gh/coconutruben/73/base 2025-12-04T09:20:54.3486947Z * [new branch] gh/coconutruben/73/head -> origin/gh/coconutruben/73/head 2025-12-04T09:20:54.3487092Z * [new branch] gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig 2025-12-04T09:20:54.3487239Z * [new branch] gh/coconutruben/74/base -> origin/gh/coconutruben/74/base 2025-12-04T09:20:54.3487602Z * [new branch] gh/coconutruben/74/head -> origin/gh/coconutruben/74/head 2025-12-04T09:20:54.3487751Z * [new branch] gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig 2025-12-04T09:20:54.3487909Z * [new branch] gh/coconutruben/79/base -> origin/gh/coconutruben/79/base 2025-12-04T09:20:54.3488118Z * [new branch] gh/coconutruben/79/head -> origin/gh/coconutruben/79/head 2025-12-04T09:20:54.3488270Z * [new branch] gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig 2025-12-04T09:20:54.3488423Z * [new branch] gh/coconutruben/80/base -> origin/gh/coconutruben/80/base 2025-12-04T09:20:54.3488567Z * [new branch] gh/coconutruben/80/head -> origin/gh/coconutruben/80/head 2025-12-04T09:20:54.3488726Z * [new branch] gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig 2025-12-04T09:20:54.3488875Z * [new branch] gh/coconutruben/82/base -> origin/gh/coconutruben/82/base 2025-12-04T09:20:54.3489026Z * [new branch] gh/coconutruben/82/head -> origin/gh/coconutruben/82/head 2025-12-04T09:20:54.3489205Z * [new branch] gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig 2025-12-04T09:20:54.3490530Z * [new branch] gh/coconutruben/83/base -> origin/gh/coconutruben/83/base 2025-12-04T09:20:54.3490864Z * [new branch] gh/coconutruben/83/head -> origin/gh/coconutruben/83/head 2025-12-04T09:20:54.3492077Z * [new branch] gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig 2025-12-04T09:20:54.3493310Z * [new branch] gh/coconutruben/84/base -> origin/gh/coconutruben/84/base 2025-12-04T09:20:54.3493736Z * [new branch] gh/coconutruben/84/head -> origin/gh/coconutruben/84/head 2025-12-04T09:20:54.3497000Z * [new branch] gh/coconutruben/84/orig -> origin/gh/coconutruben/84/orig 2025-12-04T09:20:54.3497227Z * [new branch] gh/coconutruben/85/base -> origin/gh/coconutruben/85/base 2025-12-04T09:20:54.3497396Z * [new branch] gh/coconutruben/85/head -> origin/gh/coconutruben/85/head 2025-12-04T09:20:54.3497561Z * [new branch] gh/coconutruben/85/orig -> origin/gh/coconutruben/85/orig 2025-12-04T09:20:54.3497793Z * [new branch] gh/coconutruben/86/base -> origin/gh/coconutruben/86/base 2025-12-04T09:20:54.3503034Z * [new branch] gh/coconutruben/86/head -> origin/gh/coconutruben/86/head 2025-12-04T09:20:54.3503377Z * [new branch] gh/coconutruben/86/orig -> origin/gh/coconutruben/86/orig 2025-12-04T09:20:54.3503581Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-12-04T09:20:54.3503798Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-12-04T09:20:54.3504007Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-12-04T09:20:54.3504685Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-12-04T09:20:54.3504867Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-12-04T09:20:54.3505010Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-12-04T09:20:54.3505177Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-12-04T09:20:54.3505568Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-12-04T09:20:54.3509321Z * [new branch] gh/d4l3k/1/base -> origin/gh/d4l3k/1/base 2025-12-04T09:20:54.3509495Z * [new branch] gh/d4l3k/1/head -> origin/gh/d4l3k/1/head 2025-12-04T09:20:54.3509640Z * [new branch] gh/d4l3k/2/base -> origin/gh/d4l3k/2/base 2025-12-04T09:20:54.3509784Z * [new branch] gh/d4l3k/2/head -> origin/gh/d4l3k/2/head 2025-12-04T09:20:54.3510091Z * [new branch] gh/d4l3k/2/orig -> origin/gh/d4l3k/2/orig 2025-12-04T09:20:54.3510632Z * [new branch] gh/d4l3k/3/base -> origin/gh/d4l3k/3/base 2025-12-04T09:20:54.3511548Z * [new branch] gh/d4l3k/3/head -> origin/gh/d4l3k/3/head 2025-12-04T09:20:54.3512102Z * [new branch] gh/d4l3k/3/orig -> origin/gh/d4l3k/3/orig 2025-12-04T09:20:54.3516654Z * [new branch] gh/d4l3k/4/base -> origin/gh/d4l3k/4/base 2025-12-04T09:20:54.3516831Z * [new branch] gh/d4l3k/4/head -> origin/gh/d4l3k/4/head 2025-12-04T09:20:54.3516981Z * [new branch] gh/d4l3k/4/orig -> origin/gh/d4l3k/4/orig 2025-12-04T09:20:54.3517120Z * [new branch] gh/d4l3k/5/base -> origin/gh/d4l3k/5/base 2025-12-04T09:20:54.3517251Z * [new branch] gh/d4l3k/5/orig -> origin/gh/d4l3k/5/orig 2025-12-04T09:20:54.3517467Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-12-04T09:20:54.3517908Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-12-04T09:20:54.3518834Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-12-04T09:20:54.3522522Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-12-04T09:20:54.3522896Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-12-04T09:20:54.3523157Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-12-04T09:20:54.3523336Z * [new branch] gh/desertfire/605/base -> origin/gh/desertfire/605/base 2025-12-04T09:20:54.3526096Z * [new branch] gh/desertfire/605/head -> origin/gh/desertfire/605/head 2025-12-04T09:20:54.3526310Z * [new branch] gh/desertfire/605/orig -> origin/gh/desertfire/605/orig 2025-12-04T09:20:54.3526457Z * [new branch] gh/desertfire/606/base -> origin/gh/desertfire/606/base 2025-12-04T09:20:54.3526613Z * [new branch] gh/desertfire/606/head -> origin/gh/desertfire/606/head 2025-12-04T09:20:54.3526771Z * [new branch] gh/desertfire/606/orig -> origin/gh/desertfire/606/orig 2025-12-04T09:20:54.3526920Z * [new branch] gh/desertfire/607/base -> origin/gh/desertfire/607/base 2025-12-04T09:20:54.3527453Z * [new branch] gh/desertfire/607/head -> origin/gh/desertfire/607/head 2025-12-04T09:20:54.3528378Z * [new branch] gh/desertfire/607/orig -> origin/gh/desertfire/607/orig 2025-12-04T09:20:54.3529300Z * [new branch] gh/desertfire/608/base -> origin/gh/desertfire/608/base 2025-12-04T09:20:54.3529860Z * [new branch] gh/desertfire/608/head -> origin/gh/desertfire/608/head 2025-12-04T09:20:54.3530929Z * [new branch] gh/desertfire/608/orig -> origin/gh/desertfire/608/orig 2025-12-04T09:20:54.3531951Z * [new branch] gh/desertfire/609/base -> origin/gh/desertfire/609/base 2025-12-04T09:20:54.3532264Z * [new branch] gh/desertfire/609/head -> origin/gh/desertfire/609/head 2025-12-04T09:20:54.3536897Z * [new branch] gh/desertfire/609/orig -> origin/gh/desertfire/609/orig 2025-12-04T09:20:54.3537113Z * [new branch] gh/desertfire/610/base -> origin/gh/desertfire/610/base 2025-12-04T09:20:54.3537272Z * [new branch] gh/desertfire/610/head -> origin/gh/desertfire/610/head 2025-12-04T09:20:54.3537432Z * [new branch] gh/desertfire/610/orig -> origin/gh/desertfire/610/orig 2025-12-04T09:20:54.3537589Z * [new branch] gh/desertfire/611/base -> origin/gh/desertfire/611/base 2025-12-04T09:20:54.3537895Z * [new branch] gh/desertfire/611/head -> origin/gh/desertfire/611/head 2025-12-04T09:20:54.3538285Z * [new branch] gh/desertfire/611/orig -> origin/gh/desertfire/611/orig 2025-12-04T09:20:54.3540117Z * [new branch] gh/desertfire/612/base -> origin/gh/desertfire/612/base 2025-12-04T09:20:54.3540673Z * [new branch] gh/desertfire/612/head -> origin/gh/desertfire/612/head 2025-12-04T09:20:54.3541326Z * [new branch] gh/desertfire/612/orig -> origin/gh/desertfire/612/orig 2025-12-04T09:20:54.3542585Z * [new branch] gh/desertfire/613/base -> origin/gh/desertfire/613/base 2025-12-04T09:20:54.3543051Z * [new branch] gh/desertfire/613/head -> origin/gh/desertfire/613/head 2025-12-04T09:20:54.3544012Z * [new branch] gh/desertfire/613/orig -> origin/gh/desertfire/613/orig 2025-12-04T09:20:54.3545065Z * [new branch] gh/desertfire/614/base -> origin/gh/desertfire/614/base 2025-12-04T09:20:54.3545504Z * [new branch] gh/desertfire/614/head -> origin/gh/desertfire/614/head 2025-12-04T09:20:54.3546483Z * [new branch] gh/desertfire/614/orig -> origin/gh/desertfire/614/orig 2025-12-04T09:20:54.3547356Z * [new branch] gh/desertfire/615/base -> origin/gh/desertfire/615/base 2025-12-04T09:20:54.3548451Z * [new branch] gh/desertfire/615/head -> origin/gh/desertfire/615/head 2025-12-04T09:20:54.3548739Z * [new branch] gh/desertfire/615/orig -> origin/gh/desertfire/615/orig 2025-12-04T09:20:54.3552536Z * [new branch] gh/desertfire/616/base -> origin/gh/desertfire/616/base 2025-12-04T09:20:54.3552715Z * [new branch] gh/desertfire/616/head -> origin/gh/desertfire/616/head 2025-12-04T09:20:54.3552865Z * [new branch] gh/desertfire/616/orig -> origin/gh/desertfire/616/orig 2025-12-04T09:20:54.3553007Z * [new branch] gh/desertfire/617/base -> origin/gh/desertfire/617/base 2025-12-04T09:20:54.3553178Z * [new branch] gh/desertfire/617/head -> origin/gh/desertfire/617/head 2025-12-04T09:20:54.3553545Z * [new branch] gh/desertfire/617/orig -> origin/gh/desertfire/617/orig 2025-12-04T09:20:54.3554491Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-12-04T09:20:54.3554725Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-12-04T09:20:54.3557445Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-12-04T09:20:54.3557616Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-12-04T09:20:54.3557763Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-12-04T09:20:54.3558654Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-12-04T09:20:54.3559514Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-12-04T09:20:54.3559672Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-12-04T09:20:54.3563817Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-12-04T09:20:54.3564160Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-12-04T09:20:54.3564312Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-12-04T09:20:54.3564541Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-12-04T09:20:54.3564748Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-12-04T09:20:54.3565421Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-12-04T09:20:54.3565757Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-12-04T09:20:54.3566040Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-12-04T09:20:54.3566227Z * [new branch] gh/drisspg/200/base -> origin/gh/drisspg/200/base 2025-12-04T09:20:54.3567244Z * [new branch] gh/drisspg/200/head -> origin/gh/drisspg/200/head 2025-12-04T09:20:54.3567755Z * [new branch] gh/drisspg/200/orig -> origin/gh/drisspg/200/orig 2025-12-04T09:20:54.3568791Z * [new branch] gh/drisspg/218/base -> origin/gh/drisspg/218/base 2025-12-04T09:20:54.3569063Z * [new branch] gh/drisspg/218/head -> origin/gh/drisspg/218/head 2025-12-04T09:20:54.3569997Z * [new branch] gh/drisspg/218/orig -> origin/gh/drisspg/218/orig 2025-12-04T09:20:54.3570856Z * [new branch] gh/drisspg/219/base -> origin/gh/drisspg/219/base 2025-12-04T09:20:54.3571333Z * [new branch] gh/drisspg/219/head -> origin/gh/drisspg/219/head 2025-12-04T09:20:54.3578247Z * [new branch] gh/drisspg/219/orig -> origin/gh/drisspg/219/orig 2025-12-04T09:20:54.3583428Z * [new branch] gh/drisspg/220/base -> origin/gh/drisspg/220/base 2025-12-04T09:20:54.3587646Z * [new branch] gh/drisspg/220/head -> origin/gh/drisspg/220/head 2025-12-04T09:20:54.3592463Z * [new branch] gh/drisspg/220/orig -> origin/gh/drisspg/220/orig 2025-12-04T09:20:54.3597164Z * [new branch] gh/drisspg/221/base -> origin/gh/drisspg/221/base 2025-12-04T09:20:54.3601342Z * [new branch] gh/drisspg/221/head -> origin/gh/drisspg/221/head 2025-12-04T09:20:54.3605640Z * [new branch] gh/drisspg/221/orig -> origin/gh/drisspg/221/orig 2025-12-04T09:20:54.3605811Z * [new branch] gh/drisspg/222/base -> origin/gh/drisspg/222/base 2025-12-04T09:20:54.3605973Z * [new branch] gh/drisspg/222/head -> origin/gh/drisspg/222/head 2025-12-04T09:20:54.3606157Z * [new branch] gh/drisspg/222/orig -> origin/gh/drisspg/222/orig 2025-12-04T09:20:54.3606318Z * [new branch] gh/drisspg/223/base -> origin/gh/drisspg/223/base 2025-12-04T09:20:54.3606447Z * [new branch] gh/drisspg/223/head -> origin/gh/drisspg/223/head 2025-12-04T09:20:54.3606578Z * [new branch] gh/drisspg/223/orig -> origin/gh/drisspg/223/orig 2025-12-04T09:20:54.3606714Z * [new branch] gh/drisspg/224/base -> origin/gh/drisspg/224/base 2025-12-04T09:20:54.3606841Z * [new branch] gh/drisspg/224/head -> origin/gh/drisspg/224/head 2025-12-04T09:20:54.3606975Z * [new branch] gh/drisspg/224/orig -> origin/gh/drisspg/224/orig 2025-12-04T09:20:54.3607101Z * [new branch] gh/drisspg/225/base -> origin/gh/drisspg/225/base 2025-12-04T09:20:54.3607230Z * [new branch] gh/drisspg/225/head -> origin/gh/drisspg/225/head 2025-12-04T09:20:54.3607373Z * [new branch] gh/drisspg/225/orig -> origin/gh/drisspg/225/orig 2025-12-04T09:20:54.3607515Z * [new branch] gh/drisspg/226/base -> origin/gh/drisspg/226/base 2025-12-04T09:20:54.3607657Z * [new branch] gh/drisspg/226/head -> origin/gh/drisspg/226/head 2025-12-04T09:20:54.3607809Z * [new branch] gh/drisspg/226/orig -> origin/gh/drisspg/226/orig 2025-12-04T09:20:54.3607942Z * [new branch] gh/drisspg/227/base -> origin/gh/drisspg/227/base 2025-12-04T09:20:54.3608087Z * [new branch] gh/drisspg/227/head -> origin/gh/drisspg/227/head 2025-12-04T09:20:54.3608230Z * [new branch] gh/drisspg/227/orig -> origin/gh/drisspg/227/orig 2025-12-04T09:20:54.3608371Z * [new branch] gh/drisspg/228/base -> origin/gh/drisspg/228/base 2025-12-04T09:20:54.3608519Z * [new branch] gh/drisspg/228/head -> origin/gh/drisspg/228/head 2025-12-04T09:20:54.3608853Z * [new branch] gh/drisspg/228/orig -> origin/gh/drisspg/228/orig 2025-12-04T09:20:54.3608996Z * [new branch] gh/drisspg/229/base -> origin/gh/drisspg/229/base 2025-12-04T09:20:54.3609130Z * [new branch] gh/drisspg/229/head -> origin/gh/drisspg/229/head 2025-12-04T09:20:54.3609346Z * [new branch] gh/drisspg/229/orig -> origin/gh/drisspg/229/orig 2025-12-04T09:20:54.3609491Z * [new branch] gh/drisspg/230/base -> origin/gh/drisspg/230/base 2025-12-04T09:20:54.3609624Z * [new branch] gh/drisspg/230/head -> origin/gh/drisspg/230/head 2025-12-04T09:20:54.3609758Z * [new branch] gh/drisspg/230/orig -> origin/gh/drisspg/230/orig 2025-12-04T09:20:54.3609912Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-12-04T09:20:54.3610053Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-12-04T09:20:54.3610224Z * [new branch] gh/dzmitry-huba/1/base -> origin/gh/dzmitry-huba/1/base 2025-12-04T09:20:54.3610370Z * [new branch] gh/dzmitry-huba/1/head -> origin/gh/dzmitry-huba/1/head 2025-12-04T09:20:54.3610527Z * [new branch] gh/dzmitry-huba/12/base -> origin/gh/dzmitry-huba/12/base 2025-12-04T09:20:54.3610683Z * [new branch] gh/dzmitry-huba/12/head -> origin/gh/dzmitry-huba/12/head 2025-12-04T09:20:54.3610825Z * [new branch] gh/dzmitry-huba/12/orig -> origin/gh/dzmitry-huba/12/orig 2025-12-04T09:20:54.3610973Z * [new branch] gh/dzmitry-huba/13/base -> origin/gh/dzmitry-huba/13/base 2025-12-04T09:20:54.3611112Z * [new branch] gh/dzmitry-huba/13/head -> origin/gh/dzmitry-huba/13/head 2025-12-04T09:20:54.3611252Z * [new branch] gh/dzmitry-huba/13/orig -> origin/gh/dzmitry-huba/13/orig 2025-12-04T09:20:54.3611414Z * [new branch] gh/dzmitry-huba/14/base -> origin/gh/dzmitry-huba/14/base 2025-12-04T09:20:54.3611711Z * [new branch] gh/dzmitry-huba/14/head -> origin/gh/dzmitry-huba/14/head 2025-12-04T09:20:54.3611867Z * [new branch] gh/dzmitry-huba/14/orig -> origin/gh/dzmitry-huba/14/orig 2025-12-04T09:20:54.3612011Z * [new branch] gh/dzmitry-huba/15/base -> origin/gh/dzmitry-huba/15/base 2025-12-04T09:20:54.3612151Z * [new branch] gh/dzmitry-huba/15/head -> origin/gh/dzmitry-huba/15/head 2025-12-04T09:20:54.3612298Z * [new branch] gh/dzmitry-huba/15/orig -> origin/gh/dzmitry-huba/15/orig 2025-12-04T09:20:54.3616153Z * [new branch] gh/dzmitry-huba/16/base -> origin/gh/dzmitry-huba/16/base 2025-12-04T09:20:54.3616358Z * [new branch] gh/dzmitry-huba/16/head -> origin/gh/dzmitry-huba/16/head 2025-12-04T09:20:54.3616514Z * [new branch] gh/dzmitry-huba/16/orig -> origin/gh/dzmitry-huba/16/orig 2025-12-04T09:20:54.3616698Z * [new branch] gh/dzmitry-huba/17/base -> origin/gh/dzmitry-huba/17/base 2025-12-04T09:20:54.3616856Z * [new branch] gh/dzmitry-huba/17/head -> origin/gh/dzmitry-huba/17/head 2025-12-04T09:20:54.3617019Z * [new branch] gh/dzmitry-huba/17/orig -> origin/gh/dzmitry-huba/17/orig 2025-12-04T09:20:54.3617191Z * [new branch] gh/dzmitry-huba/2/base -> origin/gh/dzmitry-huba/2/base 2025-12-04T09:20:54.3617485Z * [new branch] gh/dzmitry-huba/2/head -> origin/gh/dzmitry-huba/2/head 2025-12-04T09:20:54.3620878Z * [new branch] gh/dzmitry-huba/3/base -> origin/gh/dzmitry-huba/3/base 2025-12-04T09:20:54.3621054Z * [new branch] gh/dzmitry-huba/3/head -> origin/gh/dzmitry-huba/3/head 2025-12-04T09:20:54.3621244Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-12-04T09:20:54.3621534Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-12-04T09:20:54.3621684Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-12-04T09:20:54.3625831Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-12-04T09:20:54.3626170Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-12-04T09:20:54.3626325Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-12-04T09:20:54.3626463Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-12-04T09:20:54.3632635Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-12-04T09:20:54.3637557Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-12-04T09:20:54.3641669Z * [new branch] gh/eellison/862/base -> origin/gh/eellison/862/base 2025-12-04T09:20:54.3646703Z * [new branch] gh/eellison/862/head -> origin/gh/eellison/862/head 2025-12-04T09:20:54.3648453Z * [new branch] gh/eellison/862/orig -> origin/gh/eellison/862/orig 2025-12-04T09:20:54.3648618Z * [new branch] gh/eellison/863/base -> origin/gh/eellison/863/base 2025-12-04T09:20:54.3648788Z * [new branch] gh/eellison/863/head -> origin/gh/eellison/863/head 2025-12-04T09:20:54.3648964Z * [new branch] gh/eellison/863/orig -> origin/gh/eellison/863/orig 2025-12-04T09:20:54.3649111Z * [new branch] gh/eellison/864/base -> origin/gh/eellison/864/base 2025-12-04T09:20:54.3649256Z * [new branch] gh/eellison/864/head -> origin/gh/eellison/864/head 2025-12-04T09:20:54.3649401Z * [new branch] gh/eellison/864/orig -> origin/gh/eellison/864/orig 2025-12-04T09:20:54.3649546Z * [new branch] gh/eellison/865/base -> origin/gh/eellison/865/base 2025-12-04T09:20:54.3649698Z * [new branch] gh/eellison/865/head -> origin/gh/eellison/865/head 2025-12-04T09:20:54.3649844Z * [new branch] gh/eellison/865/orig -> origin/gh/eellison/865/orig 2025-12-04T09:20:54.3649987Z * [new branch] gh/eellison/866/base -> origin/gh/eellison/866/base 2025-12-04T09:20:54.3650140Z * [new branch] gh/eellison/866/head -> origin/gh/eellison/866/head 2025-12-04T09:20:54.3650272Z * [new branch] gh/eellison/866/orig -> origin/gh/eellison/866/orig 2025-12-04T09:20:54.3650418Z * [new branch] gh/eellison/867/base -> origin/gh/eellison/867/base 2025-12-04T09:20:54.3650572Z * [new branch] gh/eellison/867/head -> origin/gh/eellison/867/head 2025-12-04T09:20:54.3650715Z * [new branch] gh/eellison/867/orig -> origin/gh/eellison/867/orig 2025-12-04T09:20:54.3650866Z * [new branch] gh/eellison/868/base -> origin/gh/eellison/868/base 2025-12-04T09:20:54.3651010Z * [new branch] gh/eellison/868/head -> origin/gh/eellison/868/head 2025-12-04T09:20:54.3651144Z * [new branch] gh/eellison/868/orig -> origin/gh/eellison/868/orig 2025-12-04T09:20:54.3651302Z * [new branch] gh/eellison/869/base -> origin/gh/eellison/869/base 2025-12-04T09:20:54.3651603Z * [new branch] gh/eellison/869/head -> origin/gh/eellison/869/head 2025-12-04T09:20:54.3651804Z * [new branch] gh/eellison/869/orig -> origin/gh/eellison/869/orig 2025-12-04T09:20:54.3651954Z * [new branch] gh/eellison/870/base -> origin/gh/eellison/870/base 2025-12-04T09:20:54.3652103Z * [new branch] gh/eellison/870/head -> origin/gh/eellison/870/head 2025-12-04T09:20:54.3652253Z * [new branch] gh/eellison/870/orig -> origin/gh/eellison/870/orig 2025-12-04T09:20:54.3652530Z * [new branch] gh/eellison/871/base -> origin/gh/eellison/871/base 2025-12-04T09:20:54.3652693Z * [new branch] gh/eellison/871/head -> origin/gh/eellison/871/head 2025-12-04T09:20:54.3652824Z * [new branch] gh/eellison/871/orig -> origin/gh/eellison/871/orig 2025-12-04T09:20:54.3652997Z * [new branch] gh/eellison/872/base -> origin/gh/eellison/872/base 2025-12-04T09:20:54.3653139Z * [new branch] gh/eellison/872/head -> origin/gh/eellison/872/head 2025-12-04T09:20:54.3653268Z * [new branch] gh/eellison/872/orig -> origin/gh/eellison/872/orig 2025-12-04T09:20:54.3653715Z * [new branch] gh/eellison/873/base -> origin/gh/eellison/873/base 2025-12-04T09:20:54.3660160Z * [new branch] gh/eellison/873/head -> origin/gh/eellison/873/head 2025-12-04T09:20:54.3660340Z * [new branch] gh/eellison/873/orig -> origin/gh/eellison/873/orig 2025-12-04T09:20:54.3660502Z * [new branch] gh/eellison/874/base -> origin/gh/eellison/874/base 2025-12-04T09:20:54.3660653Z * [new branch] gh/eellison/874/head -> origin/gh/eellison/874/head 2025-12-04T09:20:54.3660788Z * [new branch] gh/eellison/874/orig -> origin/gh/eellison/874/orig 2025-12-04T09:20:54.3661825Z * [new branch] gh/eellison/875/base -> origin/gh/eellison/875/base 2025-12-04T09:20:54.3661985Z * [new branch] gh/eellison/875/head -> origin/gh/eellison/875/head 2025-12-04T09:20:54.3662141Z * [new branch] gh/eellison/875/orig -> origin/gh/eellison/875/orig 2025-12-04T09:20:54.3662275Z * [new branch] gh/eellison/876/base -> origin/gh/eellison/876/base 2025-12-04T09:20:54.3662549Z * [new branch] gh/eellison/876/head -> origin/gh/eellison/876/head 2025-12-04T09:20:54.3662860Z * [new branch] gh/eellison/876/orig -> origin/gh/eellison/876/orig 2025-12-04T09:20:54.3666922Z * [new branch] gh/eellison/877/base -> origin/gh/eellison/877/base 2025-12-04T09:20:54.3667249Z * [new branch] gh/eellison/877/head -> origin/gh/eellison/877/head 2025-12-04T09:20:54.3667415Z * [new branch] gh/eellison/877/orig -> origin/gh/eellison/877/orig 2025-12-04T09:20:54.3667667Z * [new branch] gh/eellison/878/base -> origin/gh/eellison/878/base 2025-12-04T09:20:54.3667835Z * [new branch] gh/eellison/878/head -> origin/gh/eellison/878/head 2025-12-04T09:20:54.3668076Z * [new branch] gh/eellison/878/orig -> origin/gh/eellison/878/orig 2025-12-04T09:20:54.3668788Z * [new branch] gh/eellison/879/base -> origin/gh/eellison/879/base 2025-12-04T09:20:54.3668973Z * [new branch] gh/eellison/879/head -> origin/gh/eellison/879/head 2025-12-04T09:20:54.3669135Z * [new branch] gh/eellison/879/orig -> origin/gh/eellison/879/orig 2025-12-04T09:20:54.3669645Z * [new branch] gh/eellison/880/base -> origin/gh/eellison/880/base 2025-12-04T09:20:54.3670826Z * [new branch] gh/eellison/880/head -> origin/gh/eellison/880/head 2025-12-04T09:20:54.3671007Z * [new branch] gh/eellison/880/orig -> origin/gh/eellison/880/orig 2025-12-04T09:20:54.3675315Z * [new branch] gh/eellison/881/base -> origin/gh/eellison/881/base 2025-12-04T09:20:54.3675651Z * [new branch] gh/eellison/881/head -> origin/gh/eellison/881/head 2025-12-04T09:20:54.3675842Z * [new branch] gh/eellison/881/orig -> origin/gh/eellison/881/orig 2025-12-04T09:20:54.3675978Z * [new branch] gh/eellison/882/base -> origin/gh/eellison/882/base 2025-12-04T09:20:54.3676112Z * [new branch] gh/eellison/882/head -> origin/gh/eellison/882/head 2025-12-04T09:20:54.3676623Z * [new branch] gh/eellison/882/orig -> origin/gh/eellison/882/orig 2025-12-04T09:20:54.3676797Z * [new branch] gh/eellison/883/base -> origin/gh/eellison/883/base 2025-12-04T09:20:54.3678208Z * [new branch] gh/eellison/883/head -> origin/gh/eellison/883/head 2025-12-04T09:20:54.3678530Z * [new branch] gh/eellison/883/orig -> origin/gh/eellison/883/orig 2025-12-04T09:20:54.3679011Z * [new branch] gh/eellison/884/base -> origin/gh/eellison/884/base 2025-12-04T09:20:54.3680838Z * [new branch] gh/eellison/884/head -> origin/gh/eellison/884/head 2025-12-04T09:20:54.3681023Z * [new branch] gh/eellison/884/orig -> origin/gh/eellison/884/orig 2025-12-04T09:20:54.3681787Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-12-04T09:20:54.3682682Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-12-04T09:20:54.3684114Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-12-04T09:20:54.3684261Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-12-04T09:20:54.3686443Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-12-04T09:20:54.3686639Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-12-04T09:20:54.3686765Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-12-04T09:20:54.3687300Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-12-04T09:20:54.3688593Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-12-04T09:20:54.3688874Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-12-04T09:20:54.3689978Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-12-04T09:20:54.3690851Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-12-04T09:20:54.3691401Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-12-04T09:20:54.3695764Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-12-04T09:20:54.3698697Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-12-04T09:20:54.3698845Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-12-04T09:20:54.3698973Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-12-04T09:20:54.3699098Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-12-04T09:20:54.3699227Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-12-04T09:20:54.3699350Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-12-04T09:20:54.3699487Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-12-04T09:20:54.3703974Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-12-04T09:20:54.3707327Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-12-04T09:20:54.3707521Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-12-04T09:20:54.3707748Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-12-04T09:20:54.3707877Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-12-04T09:20:54.3708002Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-12-04T09:20:54.3708118Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-12-04T09:20:54.3708251Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-12-04T09:20:54.3714336Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-12-04T09:20:54.3718508Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-12-04T09:20:54.3720680Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-12-04T09:20:54.3720974Z * [new branch] gh/etaf/172/base -> origin/gh/etaf/172/base 2025-12-04T09:20:54.3721130Z * [new branch] gh/etaf/172/head -> origin/gh/etaf/172/head 2025-12-04T09:20:54.3721259Z * [new branch] gh/etaf/172/orig -> origin/gh/etaf/172/orig 2025-12-04T09:20:54.3721387Z * [new branch] gh/etaf/173/base -> origin/gh/etaf/173/base 2025-12-04T09:20:54.3721523Z * [new branch] gh/etaf/173/head -> origin/gh/etaf/173/head 2025-12-04T09:20:54.3721648Z * [new branch] gh/etaf/173/orig -> origin/gh/etaf/173/orig 2025-12-04T09:20:54.3721799Z * [new branch] gh/etaf/174/base -> origin/gh/etaf/174/base 2025-12-04T09:20:54.3721935Z * [new branch] gh/etaf/174/head -> origin/gh/etaf/174/head 2025-12-04T09:20:54.3722059Z * [new branch] gh/etaf/175/base -> origin/gh/etaf/175/base 2025-12-04T09:20:54.3722194Z * [new branch] gh/etaf/175/head -> origin/gh/etaf/175/head 2025-12-04T09:20:54.3722317Z * [new branch] gh/etaf/175/orig -> origin/gh/etaf/175/orig 2025-12-04T09:20:54.3722439Z * [new branch] gh/etaf/176/base -> origin/gh/etaf/176/base 2025-12-04T09:20:54.3722569Z * [new branch] gh/etaf/176/head -> origin/gh/etaf/176/head 2025-12-04T09:20:54.3722690Z * [new branch] gh/etaf/176/orig -> origin/gh/etaf/176/orig 2025-12-04T09:20:54.3722821Z * [new branch] gh/etaf/177/base -> origin/gh/etaf/177/base 2025-12-04T09:20:54.3722954Z * [new branch] gh/etaf/177/head -> origin/gh/etaf/177/head 2025-12-04T09:20:54.3723086Z * [new branch] gh/etaf/177/orig -> origin/gh/etaf/177/orig 2025-12-04T09:20:54.3723210Z * [new branch] gh/etaf/178/base -> origin/gh/etaf/178/base 2025-12-04T09:20:54.3723328Z * [new branch] gh/etaf/178/head -> origin/gh/etaf/178/head 2025-12-04T09:20:54.3725604Z * [new branch] gh/etaf/178/orig -> origin/gh/etaf/178/orig 2025-12-04T09:20:54.3725740Z * [new branch] gh/etaf/179/base -> origin/gh/etaf/179/base 2025-12-04T09:20:54.3726013Z * [new branch] gh/etaf/179/head -> origin/gh/etaf/179/head 2025-12-04T09:20:54.3726151Z * [new branch] gh/etaf/179/orig -> origin/gh/etaf/179/orig 2025-12-04T09:20:54.3726292Z * [new branch] gh/etaf/180/base -> origin/gh/etaf/180/base 2025-12-04T09:20:54.3726430Z * [new branch] gh/etaf/180/head -> origin/gh/etaf/180/head 2025-12-04T09:20:54.3727145Z * [new branch] gh/etaf/180/orig -> origin/gh/etaf/180/orig 2025-12-04T09:20:54.3728304Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-12-04T09:20:54.3728830Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-12-04T09:20:54.3729886Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-12-04T09:20:54.3730272Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-12-04T09:20:54.3731689Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-12-04T09:20:54.3735404Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-12-04T09:20:54.3735563Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-12-04T09:20:54.3735758Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-12-04T09:20:54.3735905Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-12-04T09:20:54.3740906Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-12-04T09:20:54.3741254Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-12-04T09:20:54.3741424Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-12-04T09:20:54.3741578Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-12-04T09:20:54.3741722Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-12-04T09:20:54.3741867Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-12-04T09:20:54.3742008Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-12-04T09:20:54.3742157Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-12-04T09:20:54.3745070Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-12-04T09:20:54.3745564Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-12-04T09:20:54.3745770Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-12-04T09:20:54.3745953Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-12-04T09:20:54.3746109Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-12-04T09:20:54.3746270Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-12-04T09:20:54.3746426Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-12-04T09:20:54.3752586Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-12-04T09:20:54.3752815Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-12-04T09:20:54.3753096Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-12-04T09:20:54.3753255Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-12-04T09:20:54.3753424Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-12-04T09:20:54.3753570Z * [new branch] gh/ezyang/3144/base -> origin/gh/ezyang/3144/base 2025-12-04T09:20:54.3753725Z * [new branch] gh/ezyang/3144/head -> origin/gh/ezyang/3144/head 2025-12-04T09:20:54.3753897Z * [new branch] gh/ezyang/3144/orig -> origin/gh/ezyang/3144/orig 2025-12-04T09:20:54.3754101Z * [new branch] gh/ezyang/3167/base -> origin/gh/ezyang/3167/base 2025-12-04T09:20:54.3754261Z * [new branch] gh/ezyang/3167/head -> origin/gh/ezyang/3167/head 2025-12-04T09:20:54.3754766Z * [new branch] gh/ezyang/3167/orig -> origin/gh/ezyang/3167/orig 2025-12-04T09:20:54.3754946Z * [new branch] gh/ezyang/3173/base -> origin/gh/ezyang/3173/base 2025-12-04T09:20:54.3755092Z * [new branch] gh/ezyang/3173/head -> origin/gh/ezyang/3173/head 2025-12-04T09:20:54.3760693Z * [new branch] gh/ezyang/3173/orig -> origin/gh/ezyang/3173/orig 2025-12-04T09:20:54.3764359Z * [new branch] gh/ezyang/3175/base -> origin/gh/ezyang/3175/base 2025-12-04T09:20:54.3769323Z * [new branch] gh/ezyang/3175/head -> origin/gh/ezyang/3175/head 2025-12-04T09:20:54.3769842Z * [new branch] gh/ezyang/3175/orig -> origin/gh/ezyang/3175/orig 2025-12-04T09:20:54.3770044Z * [new branch] gh/ezyang/3182/base -> origin/gh/ezyang/3182/base 2025-12-04T09:20:54.3770352Z * [new branch] gh/ezyang/3182/head -> origin/gh/ezyang/3182/head 2025-12-04T09:20:54.3770503Z * [new branch] gh/ezyang/3182/orig -> origin/gh/ezyang/3182/orig 2025-12-04T09:20:54.3770648Z * [new branch] gh/ezyang/3185/base -> origin/gh/ezyang/3185/base 2025-12-04T09:20:54.3770857Z * [new branch] gh/ezyang/3185/head -> origin/gh/ezyang/3185/head 2025-12-04T09:20:54.3771011Z * [new branch] gh/ezyang/3185/orig -> origin/gh/ezyang/3185/orig 2025-12-04T09:20:54.3771152Z * [new branch] gh/ezyang/3189/base -> origin/gh/ezyang/3189/base 2025-12-04T09:20:54.3771301Z * [new branch] gh/ezyang/3189/head -> origin/gh/ezyang/3189/head 2025-12-04T09:20:54.3771503Z * [new branch] gh/ezyang/3189/orig -> origin/gh/ezyang/3189/orig 2025-12-04T09:20:54.3771664Z * [new branch] gh/ezyang/3191/base -> origin/gh/ezyang/3191/base 2025-12-04T09:20:54.3771819Z * [new branch] gh/ezyang/3191/head -> origin/gh/ezyang/3191/head 2025-12-04T09:20:54.3771963Z * [new branch] gh/ezyang/3191/orig -> origin/gh/ezyang/3191/orig 2025-12-04T09:20:54.3772111Z * [new branch] gh/ezyang/3192/base -> origin/gh/ezyang/3192/base 2025-12-04T09:20:54.3772453Z * [new branch] gh/ezyang/3192/head -> origin/gh/ezyang/3192/head 2025-12-04T09:20:54.3772605Z * [new branch] gh/ezyang/3192/orig -> origin/gh/ezyang/3192/orig 2025-12-04T09:20:54.3772748Z * [new branch] gh/ezyang/3193/base -> origin/gh/ezyang/3193/base 2025-12-04T09:20:54.3772918Z * [new branch] gh/ezyang/3193/head -> origin/gh/ezyang/3193/head 2025-12-04T09:20:54.3773077Z * [new branch] gh/ezyang/3193/orig -> origin/gh/ezyang/3193/orig 2025-12-04T09:20:54.3773226Z * [new branch] gh/ezyang/3194/base -> origin/gh/ezyang/3194/base 2025-12-04T09:20:54.3776765Z * [new branch] gh/ezyang/3194/head -> origin/gh/ezyang/3194/head 2025-12-04T09:20:54.3777088Z * [new branch] gh/ezyang/3194/orig -> origin/gh/ezyang/3194/orig 2025-12-04T09:20:54.3777258Z * [new branch] gh/ezyang/3195/base -> origin/gh/ezyang/3195/base 2025-12-04T09:20:54.3777577Z * [new branch] gh/ezyang/3195/head -> origin/gh/ezyang/3195/head 2025-12-04T09:20:54.3781668Z * [new branch] gh/ezyang/3195/orig -> origin/gh/ezyang/3195/orig 2025-12-04T09:20:54.3781942Z * [new branch] gh/ezyang/3196/base -> origin/gh/ezyang/3196/base 2025-12-04T09:20:54.3782087Z * [new branch] gh/ezyang/3196/head -> origin/gh/ezyang/3196/head 2025-12-04T09:20:54.3782220Z * [new branch] gh/ezyang/3196/orig -> origin/gh/ezyang/3196/orig 2025-12-04T09:20:54.3782343Z * [new branch] gh/ezyang/3197/base -> origin/gh/ezyang/3197/base 2025-12-04T09:20:54.3782482Z * [new branch] gh/ezyang/3197/head -> origin/gh/ezyang/3197/head 2025-12-04T09:20:54.3782614Z * [new branch] gh/ezyang/3197/orig -> origin/gh/ezyang/3197/orig 2025-12-04T09:20:54.3788878Z * [new branch] gh/ezyang/3198/base -> origin/gh/ezyang/3198/base 2025-12-04T09:20:54.3791032Z * [new branch] gh/ezyang/3198/head -> origin/gh/ezyang/3198/head 2025-12-04T09:20:54.3791266Z * [new branch] gh/ezyang/3198/orig -> origin/gh/ezyang/3198/orig 2025-12-04T09:20:54.3791428Z * [new branch] gh/ezyang/3199/base -> origin/gh/ezyang/3199/base 2025-12-04T09:20:54.3791585Z * [new branch] gh/ezyang/3199/head -> origin/gh/ezyang/3199/head 2025-12-04T09:20:54.3791715Z * [new branch] gh/ezyang/3199/orig -> origin/gh/ezyang/3199/orig 2025-12-04T09:20:54.3791918Z * [new branch] gh/ezyang/3200/base -> origin/gh/ezyang/3200/base 2025-12-04T09:20:54.3795195Z * [new branch] gh/ezyang/3200/head -> origin/gh/ezyang/3200/head 2025-12-04T09:20:54.3795341Z * [new branch] gh/ezyang/3200/orig -> origin/gh/ezyang/3200/orig 2025-12-04T09:20:54.3795553Z * [new branch] gh/ezyang/3201/base -> origin/gh/ezyang/3201/base 2025-12-04T09:20:54.3795773Z * [new branch] gh/ezyang/3201/head -> origin/gh/ezyang/3201/head 2025-12-04T09:20:54.3795974Z * [new branch] gh/ezyang/3201/orig -> origin/gh/ezyang/3201/orig 2025-12-04T09:20:54.3801580Z * [new branch] gh/ezyang/3202/base -> origin/gh/ezyang/3202/base 2025-12-04T09:20:54.3804569Z * [new branch] gh/ezyang/3202/head -> origin/gh/ezyang/3202/head 2025-12-04T09:20:54.3807619Z * [new branch] gh/ezyang/3202/orig -> origin/gh/ezyang/3202/orig 2025-12-04T09:20:54.3807825Z * [new branch] gh/ezyang/3203/base -> origin/gh/ezyang/3203/base 2025-12-04T09:20:54.3807965Z * [new branch] gh/ezyang/3203/head -> origin/gh/ezyang/3203/head 2025-12-04T09:20:54.3808093Z * [new branch] gh/ezyang/3203/orig -> origin/gh/ezyang/3203/orig 2025-12-04T09:20:54.3808240Z * [new branch] gh/ezyang/3204/base -> origin/gh/ezyang/3204/base 2025-12-04T09:20:54.3808375Z * [new branch] gh/ezyang/3204/head -> origin/gh/ezyang/3204/head 2025-12-04T09:20:54.3808524Z * [new branch] gh/ezyang/3204/orig -> origin/gh/ezyang/3204/orig 2025-12-04T09:20:54.3808663Z * [new branch] gh/ezyang/3205/base -> origin/gh/ezyang/3205/base 2025-12-04T09:20:54.3808803Z * [new branch] gh/ezyang/3205/head -> origin/gh/ezyang/3205/head 2025-12-04T09:20:54.3808944Z * [new branch] gh/ezyang/3205/orig -> origin/gh/ezyang/3205/orig 2025-12-04T09:20:54.3809089Z * [new branch] gh/ezyang/3206/base -> origin/gh/ezyang/3206/base 2025-12-04T09:20:54.3809236Z * [new branch] gh/ezyang/3206/head -> origin/gh/ezyang/3206/head 2025-12-04T09:20:54.3809378Z * [new branch] gh/ezyang/3206/orig -> origin/gh/ezyang/3206/orig 2025-12-04T09:20:54.3809521Z * [new branch] gh/ezyang/3207/base -> origin/gh/ezyang/3207/base 2025-12-04T09:20:54.3809665Z * [new branch] gh/ezyang/3207/head -> origin/gh/ezyang/3207/head 2025-12-04T09:20:54.3809804Z * [new branch] gh/ezyang/3207/orig -> origin/gh/ezyang/3207/orig 2025-12-04T09:20:54.3809942Z * [new branch] gh/ezyang/3208/base -> origin/gh/ezyang/3208/base 2025-12-04T09:20:54.3810089Z * [new branch] gh/ezyang/3208/head -> origin/gh/ezyang/3208/head 2025-12-04T09:20:54.3810229Z * [new branch] gh/ezyang/3208/orig -> origin/gh/ezyang/3208/orig 2025-12-04T09:20:54.3810377Z * [new branch] gh/ezyang/3209/base -> origin/gh/ezyang/3209/base 2025-12-04T09:20:54.3810519Z * [new branch] gh/ezyang/3209/head -> origin/gh/ezyang/3209/head 2025-12-04T09:20:54.3810658Z * [new branch] gh/ezyang/3209/orig -> origin/gh/ezyang/3209/orig 2025-12-04T09:20:54.3811200Z * [new branch] gh/fadara01/3/base -> origin/gh/fadara01/3/base 2025-12-04T09:20:54.3811346Z * [new branch] gh/fadara01/3/head -> origin/gh/fadara01/3/head 2025-12-04T09:20:54.3811656Z * [new branch] gh/fadara01/3/orig -> origin/gh/fadara01/3/orig 2025-12-04T09:20:54.3811817Z * [new branch] gh/fadara01/5/base -> origin/gh/fadara01/5/base 2025-12-04T09:20:54.3811970Z * [new branch] gh/fadara01/5/head -> origin/gh/fadara01/5/head 2025-12-04T09:20:54.3812128Z * [new branch] gh/fadara01/5/orig -> origin/gh/fadara01/5/orig 2025-12-04T09:20:54.3819548Z * [new branch] gh/fadara01/6/base -> origin/gh/fadara01/6/base 2025-12-04T09:20:54.3819786Z * [new branch] gh/fadara01/6/head -> origin/gh/fadara01/6/head 2025-12-04T09:20:54.3819996Z * [new branch] gh/fadara01/6/orig -> origin/gh/fadara01/6/orig 2025-12-04T09:20:54.3820204Z * [new branch] gh/fadara01/7/base -> origin/gh/fadara01/7/base 2025-12-04T09:20:54.3820410Z * [new branch] gh/fadara01/7/head -> origin/gh/fadara01/7/head 2025-12-04T09:20:54.3825551Z * [new branch] gh/fadara01/7/orig -> origin/gh/fadara01/7/orig 2025-12-04T09:20:54.3830116Z * [new branch] gh/fadara01/8/base -> origin/gh/fadara01/8/base 2025-12-04T09:20:54.3830281Z * [new branch] gh/fadara01/8/head -> origin/gh/fadara01/8/head 2025-12-04T09:20:54.3830440Z * [new branch] gh/fadara01/8/orig -> origin/gh/fadara01/8/orig 2025-12-04T09:20:54.3830585Z * [new branch] gh/fadara01/9/base -> origin/gh/fadara01/9/base 2025-12-04T09:20:54.3830721Z * [new branch] gh/fadara01/9/head -> origin/gh/fadara01/9/head 2025-12-04T09:20:54.3830849Z * [new branch] gh/fadara01/9/orig -> origin/gh/fadara01/9/orig 2025-12-04T09:20:54.3831005Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-12-04T09:20:54.3831133Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-12-04T09:20:54.3831259Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-12-04T09:20:54.3831390Z * [new branch] gh/fduwjj/211/base -> origin/gh/fduwjj/211/base 2025-12-04T09:20:54.3831512Z * [new branch] gh/fduwjj/211/head -> origin/gh/fduwjj/211/head 2025-12-04T09:20:54.3831636Z * [new branch] gh/fduwjj/211/orig -> origin/gh/fduwjj/211/orig 2025-12-04T09:20:54.3831773Z * [new branch] gh/fduwjj/212/base -> origin/gh/fduwjj/212/base 2025-12-04T09:20:54.3831895Z * [new branch] gh/fduwjj/212/head -> origin/gh/fduwjj/212/head 2025-12-04T09:20:54.3832026Z * [new branch] gh/fduwjj/212/orig -> origin/gh/fduwjj/212/orig 2025-12-04T09:20:54.3832151Z * [new branch] gh/fduwjj/213/base -> origin/gh/fduwjj/213/base 2025-12-04T09:20:54.3832275Z * [new branch] gh/fduwjj/213/head -> origin/gh/fduwjj/213/head 2025-12-04T09:20:54.3832406Z * [new branch] gh/fduwjj/213/orig -> origin/gh/fduwjj/213/orig 2025-12-04T09:20:54.3832527Z * [new branch] gh/fduwjj/226/base -> origin/gh/fduwjj/226/base 2025-12-04T09:20:54.3832670Z * [new branch] gh/fduwjj/226/head -> origin/gh/fduwjj/226/head 2025-12-04T09:20:54.3832885Z * [new branch] gh/fduwjj/226/orig -> origin/gh/fduwjj/226/orig 2025-12-04T09:20:54.3838810Z * [new branch] gh/fduwjj/229/base -> origin/gh/fduwjj/229/base 2025-12-04T09:20:54.3839481Z * [new branch] gh/fduwjj/229/head -> origin/gh/fduwjj/229/head 2025-12-04T09:20:54.3839871Z * [new branch] gh/fduwjj/229/orig -> origin/gh/fduwjj/229/orig 2025-12-04T09:20:54.3840044Z * [new branch] gh/fduwjj/233/base -> origin/gh/fduwjj/233/base 2025-12-04T09:20:54.3840200Z * [new branch] gh/fduwjj/233/head -> origin/gh/fduwjj/233/head 2025-12-04T09:20:54.3840351Z * [new branch] gh/fduwjj/233/orig -> origin/gh/fduwjj/233/orig 2025-12-04T09:20:54.3840650Z * [new branch] gh/fduwjj/234/base -> origin/gh/fduwjj/234/base 2025-12-04T09:20:54.3841009Z * [new branch] gh/fduwjj/234/head -> origin/gh/fduwjj/234/head 2025-12-04T09:20:54.3841174Z * [new branch] gh/fduwjj/234/orig -> origin/gh/fduwjj/234/orig 2025-12-04T09:20:54.3841582Z * [new branch] gh/fduwjj/235/base -> origin/gh/fduwjj/235/base 2025-12-04T09:20:54.3847360Z * [new branch] gh/fduwjj/235/head -> origin/gh/fduwjj/235/head 2025-12-04T09:20:54.3847560Z * [new branch] gh/fduwjj/235/orig -> origin/gh/fduwjj/235/orig 2025-12-04T09:20:54.3847890Z * [new branch] gh/fduwjj/236/base -> origin/gh/fduwjj/236/base 2025-12-04T09:20:54.3848048Z * [new branch] gh/fduwjj/236/head -> origin/gh/fduwjj/236/head 2025-12-04T09:20:54.3848201Z * [new branch] gh/fduwjj/236/orig -> origin/gh/fduwjj/236/orig 2025-12-04T09:20:54.3848370Z * [new branch] gh/fduwjj/237/base -> origin/gh/fduwjj/237/base 2025-12-04T09:20:54.3848518Z * [new branch] gh/fduwjj/237/head -> origin/gh/fduwjj/237/head 2025-12-04T09:20:54.3848663Z * [new branch] gh/fduwjj/237/orig -> origin/gh/fduwjj/237/orig 2025-12-04T09:20:54.3848837Z * [new branch] gh/fduwjj/238/base -> origin/gh/fduwjj/238/base 2025-12-04T09:20:54.3848982Z * [new branch] gh/fduwjj/238/head -> origin/gh/fduwjj/238/head 2025-12-04T09:20:54.3849125Z * [new branch] gh/fduwjj/238/orig -> origin/gh/fduwjj/238/orig 2025-12-04T09:20:54.3849277Z * [new branch] gh/fduwjj/239/base -> origin/gh/fduwjj/239/base 2025-12-04T09:20:54.3849414Z * [new branch] gh/fduwjj/239/head -> origin/gh/fduwjj/239/head 2025-12-04T09:20:54.3849556Z * [new branch] gh/fduwjj/239/orig -> origin/gh/fduwjj/239/orig 2025-12-04T09:20:54.3849746Z * [new branch] gh/fegin/332/base -> origin/gh/fegin/332/base 2025-12-04T09:20:54.3849896Z * [new branch] gh/fegin/332/head -> origin/gh/fegin/332/head 2025-12-04T09:20:54.3850045Z * [new branch] gh/fegin/332/orig -> origin/gh/fegin/332/orig 2025-12-04T09:20:54.3850693Z * [new branch] gh/fegin/333/base -> origin/gh/fegin/333/base 2025-12-04T09:20:54.3851740Z * [new branch] gh/fegin/333/head -> origin/gh/fegin/333/head 2025-12-04T09:20:54.3852243Z * [new branch] gh/fegin/333/orig -> origin/gh/fegin/333/orig 2025-12-04T09:20:54.3856979Z * [new branch] gh/fegin/334/base -> origin/gh/fegin/334/base 2025-12-04T09:20:54.3857508Z * [new branch] gh/fegin/334/head -> origin/gh/fegin/334/head 2025-12-04T09:20:54.3857667Z * [new branch] gh/fegin/334/orig -> origin/gh/fegin/334/orig 2025-12-04T09:20:54.3857818Z * [new branch] gh/fegin/335/base -> origin/gh/fegin/335/base 2025-12-04T09:20:54.3857959Z * [new branch] gh/fegin/335/head -> origin/gh/fegin/335/head 2025-12-04T09:20:54.3858101Z * [new branch] gh/fegin/335/orig -> origin/gh/fegin/335/orig 2025-12-04T09:20:54.3858359Z * [new branch] gh/fffrog/160/base -> origin/gh/fffrog/160/base 2025-12-04T09:20:54.3859716Z * [new branch] gh/fffrog/160/head -> origin/gh/fffrog/160/head 2025-12-04T09:20:54.3860071Z * [new branch] gh/fffrog/177/base -> origin/gh/fffrog/177/base 2025-12-04T09:20:54.3860546Z * [new branch] gh/fffrog/177/head -> origin/gh/fffrog/177/head 2025-12-04T09:20:54.3863435Z * [new branch] gh/fffrog/177/orig -> origin/gh/fffrog/177/orig 2025-12-04T09:20:54.3863740Z * [new branch] gh/fffrog/178/base -> origin/gh/fffrog/178/base 2025-12-04T09:20:54.3864088Z * [new branch] gh/fffrog/178/head -> origin/gh/fffrog/178/head 2025-12-04T09:20:54.3864383Z * [new branch] gh/fffrog/178/orig -> origin/gh/fffrog/178/orig 2025-12-04T09:20:54.3864571Z * [new branch] gh/fffrog/181/base -> origin/gh/fffrog/181/base 2025-12-04T09:20:54.3865192Z * [new branch] gh/fffrog/181/head -> origin/gh/fffrog/181/head 2025-12-04T09:20:54.3868132Z * [new branch] gh/fffrog/181/orig -> origin/gh/fffrog/181/orig 2025-12-04T09:20:54.3868633Z * [new branch] gh/fffrog/183/base -> origin/gh/fffrog/183/base 2025-12-04T09:20:54.3868930Z * [new branch] gh/fffrog/183/head -> origin/gh/fffrog/183/head 2025-12-04T09:20:54.3869267Z * [new branch] gh/fffrog/183/orig -> origin/gh/fffrog/183/orig 2025-12-04T09:20:54.3869698Z * [new branch] gh/fxdawnn/10/base -> origin/gh/fxdawnn/10/base 2025-12-04T09:20:54.3869899Z * [new branch] gh/fxdawnn/10/head -> origin/gh/fxdawnn/10/head 2025-12-04T09:20:54.3871191Z * [new branch] gh/fxdawnn/10/orig -> origin/gh/fxdawnn/10/orig 2025-12-04T09:20:54.3871582Z * [new branch] gh/fxdawnn/11/base -> origin/gh/fxdawnn/11/base 2025-12-04T09:20:54.3874596Z * [new branch] gh/fxdawnn/11/head -> origin/gh/fxdawnn/11/head 2025-12-04T09:20:54.3874918Z * [new branch] gh/fxdawnn/11/orig -> origin/gh/fxdawnn/11/orig 2025-12-04T09:20:54.3875217Z * [new branch] gh/fxdawnn/12/base -> origin/gh/fxdawnn/12/base 2025-12-04T09:20:54.3875363Z * [new branch] gh/fxdawnn/12/head -> origin/gh/fxdawnn/12/head 2025-12-04T09:20:54.3875512Z * [new branch] gh/fxdawnn/12/orig -> origin/gh/fxdawnn/12/orig 2025-12-04T09:20:54.3877613Z * [new branch] gh/fxdawnn/13/base -> origin/gh/fxdawnn/13/base 2025-12-04T09:20:54.3877939Z * [new branch] gh/fxdawnn/13/head -> origin/gh/fxdawnn/13/head 2025-12-04T09:20:54.3878244Z * [new branch] gh/fxdawnn/13/orig -> origin/gh/fxdawnn/13/orig 2025-12-04T09:20:54.3878582Z * [new branch] gh/fxdawnn/14/base -> origin/gh/fxdawnn/14/base 2025-12-04T09:20:54.3880718Z * [new branch] gh/fxdawnn/14/head -> origin/gh/fxdawnn/14/head 2025-12-04T09:20:54.3880896Z * [new branch] gh/fxdawnn/14/orig -> origin/gh/fxdawnn/14/orig 2025-12-04T09:20:54.3881050Z * [new branch] gh/fxdawnn/15/base -> origin/gh/fxdawnn/15/base 2025-12-04T09:20:54.3881373Z * [new branch] gh/fxdawnn/15/head -> origin/gh/fxdawnn/15/head 2025-12-04T09:20:54.3882886Z * [new branch] gh/fxdawnn/15/orig -> origin/gh/fxdawnn/15/orig 2025-12-04T09:20:54.3883068Z * [new branch] gh/fxdawnn/6/base -> origin/gh/fxdawnn/6/base 2025-12-04T09:20:54.3883848Z * [new branch] gh/fxdawnn/6/head -> origin/gh/fxdawnn/6/head 2025-12-04T09:20:54.3884234Z * [new branch] gh/fxdawnn/6/orig -> origin/gh/fxdawnn/6/orig 2025-12-04T09:20:54.3887301Z * [new branch] gh/fxdawnn/7/base -> origin/gh/fxdawnn/7/base 2025-12-04T09:20:54.3887485Z * [new branch] gh/fxdawnn/7/head -> origin/gh/fxdawnn/7/head 2025-12-04T09:20:54.3887641Z * [new branch] gh/fxdawnn/7/orig -> origin/gh/fxdawnn/7/orig 2025-12-04T09:20:54.3887815Z * [new branch] gh/fxdawnn/9/base -> origin/gh/fxdawnn/9/base 2025-12-04T09:20:54.3888381Z * [new branch] gh/fxdawnn/9/head -> origin/gh/fxdawnn/9/head 2025-12-04T09:20:54.3889553Z * [new branch] gh/fxdawnn/9/orig -> origin/gh/fxdawnn/9/orig 2025-12-04T09:20:54.3890741Z * [new branch] gh/galv/1/base -> origin/gh/galv/1/base 2025-12-04T09:20:54.3891149Z * [new branch] gh/galv/1/head -> origin/gh/galv/1/head 2025-12-04T09:20:54.3892280Z * [new branch] gh/galv/1/orig -> origin/gh/galv/1/orig 2025-12-04T09:20:54.3892925Z * [new branch] gh/galv/2/base -> origin/gh/galv/2/base 2025-12-04T09:20:54.3895639Z * [new branch] gh/galv/2/head -> origin/gh/galv/2/head 2025-12-04T09:20:54.3899983Z * [new branch] gh/galv/2/orig -> origin/gh/galv/2/orig 2025-12-04T09:20:54.3904468Z * [new branch] gh/galv/3/base -> origin/gh/galv/3/base 2025-12-04T09:20:54.3909763Z * [new branch] gh/galv/3/head -> origin/gh/galv/3/head 2025-12-04T09:20:54.3912484Z * [new branch] gh/galv/3/orig -> origin/gh/galv/3/orig 2025-12-04T09:20:54.3912756Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-12-04T09:20:54.3917466Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-12-04T09:20:54.3919814Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-12-04T09:20:54.3920131Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-12-04T09:20:54.3920313Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-12-04T09:20:54.3920491Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-12-04T09:20:54.3920646Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-12-04T09:20:54.3920785Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-12-04T09:20:54.3921007Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-12-04T09:20:54.3921278Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-12-04T09:20:54.3921427Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-12-04T09:20:54.3921554Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-12-04T09:20:54.3921693Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-12-04T09:20:54.3921839Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-12-04T09:20:54.3921969Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-12-04T09:20:54.3922112Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-12-04T09:20:54.3922239Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-12-04T09:20:54.3922374Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-12-04T09:20:54.3922501Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-12-04T09:20:54.3922631Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-12-04T09:20:54.3922776Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-12-04T09:20:54.3922905Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-12-04T09:20:54.3923039Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-12-04T09:20:54.3923172Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-12-04T09:20:54.3923306Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-12-04T09:20:54.3923441Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-12-04T09:20:54.3923568Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-12-04T09:20:54.3923702Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-12-04T09:20:54.3923829Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-12-04T09:20:54.3924087Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-12-04T09:20:54.3924227Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-12-04T09:20:54.3924359Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-12-04T09:20:54.3924526Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-12-04T09:20:54.3924673Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-12-04T09:20:54.3924805Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-12-04T09:20:54.3924961Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-12-04T09:20:54.3926157Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-12-04T09:20:54.3926350Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-12-04T09:20:54.3927511Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-12-04T09:20:54.3928757Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-12-04T09:20:54.3929073Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-12-04T09:20:54.3929520Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-12-04T09:20:54.3930760Z * [new branch] gh/guangyey/208/base -> origin/gh/guangyey/208/base 2025-12-04T09:20:54.3931173Z * [new branch] gh/guangyey/208/head -> origin/gh/guangyey/208/head 2025-12-04T09:20:54.3932189Z * [new branch] gh/guangyey/208/orig -> origin/gh/guangyey/208/orig 2025-12-04T09:20:54.3933655Z * [new branch] gh/guangyey/228/base -> origin/gh/guangyey/228/base 2025-12-04T09:20:54.3933849Z * [new branch] gh/guangyey/228/head -> origin/gh/guangyey/228/head 2025-12-04T09:20:54.3937077Z * [new branch] gh/guangyey/228/orig -> origin/gh/guangyey/228/orig 2025-12-04T09:20:54.3937265Z * [new branch] gh/guangyey/230/base -> origin/gh/guangyey/230/base 2025-12-04T09:20:54.3937414Z * [new branch] gh/guangyey/230/head -> origin/gh/guangyey/230/head 2025-12-04T09:20:54.3937600Z * [new branch] gh/guangyey/230/orig -> origin/gh/guangyey/230/orig 2025-12-04T09:20:54.3942275Z * [new branch] gh/guangyey/231/base -> origin/gh/guangyey/231/base 2025-12-04T09:20:54.3946660Z * [new branch] gh/guangyey/231/head -> origin/gh/guangyey/231/head 2025-12-04T09:20:54.3951453Z * [new branch] gh/guangyey/231/orig -> origin/gh/guangyey/231/orig 2025-12-04T09:20:54.3956294Z * [new branch] gh/guangyey/232/base -> origin/gh/guangyey/232/base 2025-12-04T09:20:54.3961088Z * [new branch] gh/guangyey/232/head -> origin/gh/guangyey/232/head 2025-12-04T09:20:54.3965763Z * [new branch] gh/guangyey/232/orig -> origin/gh/guangyey/232/orig 2025-12-04T09:20:54.3967934Z * [new branch] gh/guangyey/233/base -> origin/gh/guangyey/233/base 2025-12-04T09:20:54.3968102Z * [new branch] gh/guangyey/233/head -> origin/gh/guangyey/233/head 2025-12-04T09:20:54.3968542Z * [new branch] gh/guangyey/233/orig -> origin/gh/guangyey/233/orig 2025-12-04T09:20:54.3968697Z * [new branch] gh/guangyey/234/base -> origin/gh/guangyey/234/base 2025-12-04T09:20:54.3968939Z * [new branch] gh/guangyey/234/head -> origin/gh/guangyey/234/head 2025-12-04T09:20:54.3969085Z * [new branch] gh/guangyey/234/orig -> origin/gh/guangyey/234/orig 2025-12-04T09:20:54.3969253Z * [new branch] gh/guangyey/235/base -> origin/gh/guangyey/235/base 2025-12-04T09:20:54.3969551Z * [new branch] gh/guangyey/235/head -> origin/gh/guangyey/235/head 2025-12-04T09:20:54.3969684Z * [new branch] gh/guangyey/235/orig -> origin/gh/guangyey/235/orig 2025-12-04T09:20:54.3969813Z * [new branch] gh/guangyey/236/base -> origin/gh/guangyey/236/base 2025-12-04T09:20:54.3969999Z * [new branch] gh/guangyey/236/head -> origin/gh/guangyey/236/head 2025-12-04T09:20:54.3970134Z * [new branch] gh/guangyey/236/orig -> origin/gh/guangyey/236/orig 2025-12-04T09:20:54.3970274Z * [new branch] gh/guangyey/237/base -> origin/gh/guangyey/237/base 2025-12-04T09:20:54.3970407Z * [new branch] gh/guangyey/237/head -> origin/gh/guangyey/237/head 2025-12-04T09:20:54.3970538Z * [new branch] gh/guangyey/237/orig -> origin/gh/guangyey/237/orig 2025-12-04T09:20:54.3970677Z * [new branch] gh/guangyey/238/base -> origin/gh/guangyey/238/base 2025-12-04T09:20:54.3970813Z * [new branch] gh/guangyey/238/head -> origin/gh/guangyey/238/head 2025-12-04T09:20:54.3970953Z * [new branch] gh/guangyey/239/base -> origin/gh/guangyey/239/base 2025-12-04T09:20:54.3971100Z * [new branch] gh/guangyey/239/head -> origin/gh/guangyey/239/head 2025-12-04T09:20:54.3971234Z * [new branch] gh/guangyey/239/orig -> origin/gh/guangyey/239/orig 2025-12-04T09:20:54.3971373Z * [new branch] gh/guangyey/240/base -> origin/gh/guangyey/240/base 2025-12-04T09:20:54.3971748Z * [new branch] gh/guangyey/240/head -> origin/gh/guangyey/240/head 2025-12-04T09:20:54.3971888Z * [new branch] gh/guangyey/240/orig -> origin/gh/guangyey/240/orig 2025-12-04T09:20:54.3972025Z * [new branch] gh/guangyey/241/base -> origin/gh/guangyey/241/base 2025-12-04T09:20:54.3972150Z * [new branch] gh/guangyey/241/head -> origin/gh/guangyey/241/head 2025-12-04T09:20:54.3972444Z * [new branch] gh/guangyey/241/orig -> origin/gh/guangyey/241/orig 2025-12-04T09:20:54.3972598Z * [new branch] gh/guangyey/242/base -> origin/gh/guangyey/242/base 2025-12-04T09:20:54.3972731Z * [new branch] gh/guangyey/242/head -> origin/gh/guangyey/242/head 2025-12-04T09:20:54.3972879Z * [new branch] gh/guangyey/242/orig -> origin/gh/guangyey/242/orig 2025-12-04T09:20:54.3973013Z * [new branch] gh/guangyey/243/base -> origin/gh/guangyey/243/base 2025-12-04T09:20:54.3973166Z * [new branch] gh/guangyey/243/head -> origin/gh/guangyey/243/head 2025-12-04T09:20:54.3973296Z * [new branch] gh/guangyey/243/orig -> origin/gh/guangyey/243/orig 2025-12-04T09:20:54.3973421Z * [new branch] gh/guangyey/244/base -> origin/gh/guangyey/244/base 2025-12-04T09:20:54.3973556Z * [new branch] gh/guangyey/244/head -> origin/gh/guangyey/244/head 2025-12-04T09:20:54.3973686Z * [new branch] gh/guangyey/244/orig -> origin/gh/guangyey/244/orig 2025-12-04T09:20:54.3973825Z * [new branch] gh/guangyey/245/base -> origin/gh/guangyey/245/base 2025-12-04T09:20:54.3973954Z * [new branch] gh/guangyey/245/head -> origin/gh/guangyey/245/head 2025-12-04T09:20:54.3974082Z * [new branch] gh/guangyey/245/orig -> origin/gh/guangyey/245/orig 2025-12-04T09:20:54.3976679Z * [new branch] gh/guangyey/246/base -> origin/gh/guangyey/246/base 2025-12-04T09:20:54.3982923Z * [new branch] gh/guangyey/246/head -> origin/gh/guangyey/246/head 2025-12-04T09:20:54.3987346Z * [new branch] gh/guangyey/246/orig -> origin/gh/guangyey/246/orig 2025-12-04T09:20:54.3989665Z * [new branch] gh/guangyey/247/base -> origin/gh/guangyey/247/base 2025-12-04T09:20:54.3989846Z * [new branch] gh/guangyey/247/head -> origin/gh/guangyey/247/head 2025-12-04T09:20:54.3990194Z * [new branch] gh/guangyey/247/orig -> origin/gh/guangyey/247/orig 2025-12-04T09:20:54.3990327Z * [new branch] gh/guangyey/248/base -> origin/gh/guangyey/248/base 2025-12-04T09:20:54.3990530Z * [new branch] gh/guangyey/248/head -> origin/gh/guangyey/248/head 2025-12-04T09:20:54.3990714Z * [new branch] gh/guangyey/248/orig -> origin/gh/guangyey/248/orig 2025-12-04T09:20:54.3990887Z * [new branch] gh/guangyey/249/base -> origin/gh/guangyey/249/base 2025-12-04T09:20:54.3991021Z * [new branch] gh/guangyey/249/head -> origin/gh/guangyey/249/head 2025-12-04T09:20:54.3991150Z * [new branch] gh/guangyey/249/orig -> origin/gh/guangyey/249/orig 2025-12-04T09:20:54.3991288Z * [new branch] gh/guangyey/250/base -> origin/gh/guangyey/250/base 2025-12-04T09:20:54.3991422Z * [new branch] gh/guangyey/250/head -> origin/gh/guangyey/250/head 2025-12-04T09:20:54.3991558Z * [new branch] gh/guangyey/250/orig -> origin/gh/guangyey/250/orig 2025-12-04T09:20:54.3991687Z * [new branch] gh/guangyey/251/base -> origin/gh/guangyey/251/base 2025-12-04T09:20:54.3991819Z * [new branch] gh/guangyey/251/head -> origin/gh/guangyey/251/head 2025-12-04T09:20:54.3991954Z * [new branch] gh/guangyey/251/orig -> origin/gh/guangyey/251/orig 2025-12-04T09:20:54.3992090Z * [new branch] gh/guangyey/252/base -> origin/gh/guangyey/252/base 2025-12-04T09:20:54.3992220Z * [new branch] gh/guangyey/252/head -> origin/gh/guangyey/252/head 2025-12-04T09:20:54.3992344Z * [new branch] gh/guangyey/252/orig -> origin/gh/guangyey/252/orig 2025-12-04T09:20:54.3992468Z * [new branch] gh/guangyey/253/base -> origin/gh/guangyey/253/base 2025-12-04T09:20:54.3992603Z * [new branch] gh/guangyey/253/head -> origin/gh/guangyey/253/head 2025-12-04T09:20:54.3992730Z * [new branch] gh/guangyey/253/orig -> origin/gh/guangyey/253/orig 2025-12-04T09:20:54.3996975Z * [new branch] gh/guangyey/254/base -> origin/gh/guangyey/254/base 2025-12-04T09:20:54.3999274Z * [new branch] gh/guangyey/254/head -> origin/gh/guangyey/254/head 2025-12-04T09:20:54.4004456Z * [new branch] gh/guangyey/254/orig -> origin/gh/guangyey/254/orig 2025-12-04T09:20:54.4008651Z * [new branch] gh/guangyey/255/base -> origin/gh/guangyey/255/base 2025-12-04T09:20:54.4013372Z * [new branch] gh/guangyey/255/head -> origin/gh/guangyey/255/head 2025-12-04T09:20:54.4017786Z * [new branch] gh/guangyey/255/orig -> origin/gh/guangyey/255/orig 2025-12-04T09:20:54.4022069Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-12-04T09:20:54.4027584Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-12-04T09:20:54.4027961Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-12-04T09:20:54.4028147Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-12-04T09:20:54.4028299Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-12-04T09:20:54.4028452Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-12-04T09:20:54.4028613Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-12-04T09:20:54.4028763Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-12-04T09:20:54.4028924Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-12-04T09:20:54.4029218Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-12-04T09:20:54.4029374Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-12-04T09:20:54.4029535Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-12-04T09:20:54.4029728Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-12-04T09:20:54.4029892Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-12-04T09:20:54.4030047Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-12-04T09:20:54.4030200Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-12-04T09:20:54.4030366Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-12-04T09:20:54.4030520Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-12-04T09:20:54.4030675Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-12-04T09:20:54.4030821Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-12-04T09:20:54.4030974Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-12-04T09:20:54.4031131Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-12-04T09:20:54.4031277Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-12-04T09:20:54.4031426Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-12-04T09:20:54.4031580Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-12-04T09:20:54.4031726Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-12-04T09:20:54.4031881Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-12-04T09:20:54.4032027Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-12-04T09:20:54.4032178Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-12-04T09:20:54.4032334Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-12-04T09:20:54.4032481Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-12-04T09:20:54.4032639Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-12-04T09:20:54.4032788Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-12-04T09:20:54.4032934Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-12-04T09:20:54.4033091Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-12-04T09:20:54.4033239Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-12-04T09:20:54.4033396Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-12-04T09:20:54.4033543Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-12-04T09:20:54.4033690Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-12-04T09:20:54.4033845Z * [new branch] gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base 2025-12-04T09:20:54.4033998Z * [new branch] gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head 2025-12-04T09:20:54.4034152Z * [new branch] gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig 2025-12-04T09:20:54.4034342Z * [new branch] gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base 2025-12-04T09:20:54.4034491Z * [new branch] gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head 2025-12-04T09:20:54.4034645Z * [new branch] gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig 2025-12-04T09:20:54.4034851Z * [new branch] gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base 2025-12-04T09:20:54.4035013Z * [new branch] gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head 2025-12-04T09:20:54.4035588Z * [new branch] gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig 2025-12-04T09:20:54.4035740Z * [new branch] gh/guilhermeleobas/253/base -> origin/gh/guilhermeleobas/253/base 2025-12-04T09:20:54.4035892Z * [new branch] gh/guilhermeleobas/253/head -> origin/gh/guilhermeleobas/253/head 2025-12-04T09:20:54.4052751Z * [new branch] gh/guilhermeleobas/253/orig -> origin/gh/guilhermeleobas/253/orig 2025-12-04T09:20:54.4053818Z * [new branch] gh/guilhermeleobas/254/base -> origin/gh/guilhermeleobas/254/base 2025-12-04T09:20:54.4054074Z * [new branch] gh/guilhermeleobas/254/head -> origin/gh/guilhermeleobas/254/head 2025-12-04T09:20:54.4054266Z * [new branch] gh/guilhermeleobas/254/orig -> origin/gh/guilhermeleobas/254/orig 2025-12-04T09:20:54.4054501Z * [new branch] gh/guilhermeleobas/255/base -> origin/gh/guilhermeleobas/255/base 2025-12-04T09:20:54.4054771Z * [new branch] gh/guilhermeleobas/255/head -> origin/gh/guilhermeleobas/255/head 2025-12-04T09:20:54.4055012Z * [new branch] gh/guilhermeleobas/255/orig -> origin/gh/guilhermeleobas/255/orig 2025-12-04T09:20:54.4055185Z * [new branch] gh/guilhermeleobas/256/base -> origin/gh/guilhermeleobas/256/base 2025-12-04T09:20:54.4055346Z * [new branch] gh/guilhermeleobas/256/head -> origin/gh/guilhermeleobas/256/head 2025-12-04T09:20:54.4055607Z * [new branch] gh/guilhermeleobas/256/orig -> origin/gh/guilhermeleobas/256/orig 2025-12-04T09:20:54.4055806Z * [new branch] gh/guilhermeleobas/257/base -> origin/gh/guilhermeleobas/257/base 2025-12-04T09:20:54.4056032Z * [new branch] gh/guilhermeleobas/257/head -> origin/gh/guilhermeleobas/257/head 2025-12-04T09:20:54.4060770Z * [new branch] gh/guilhermeleobas/257/orig -> origin/gh/guilhermeleobas/257/orig 2025-12-04T09:20:54.4063514Z * [new branch] gh/guilhermeleobas/258/base -> origin/gh/guilhermeleobas/258/base 2025-12-04T09:20:54.4064059Z * [new branch] gh/guilhermeleobas/258/head -> origin/gh/guilhermeleobas/258/head 2025-12-04T09:20:54.4064268Z * [new branch] gh/guilhermeleobas/258/orig -> origin/gh/guilhermeleobas/258/orig 2025-12-04T09:20:54.4064420Z * [new branch] gh/guilhermeleobas/259/base -> origin/gh/guilhermeleobas/259/base 2025-12-04T09:20:54.4064585Z * [new branch] gh/guilhermeleobas/259/head -> origin/gh/guilhermeleobas/259/head 2025-12-04T09:20:54.4064745Z * [new branch] gh/guilhermeleobas/259/orig -> origin/gh/guilhermeleobas/259/orig 2025-12-04T09:20:54.4064896Z * [new branch] gh/guilhermeleobas/260/base -> origin/gh/guilhermeleobas/260/base 2025-12-04T09:20:54.4065062Z * [new branch] gh/guilhermeleobas/260/head -> origin/gh/guilhermeleobas/260/head 2025-12-04T09:20:54.4065213Z * [new branch] gh/guilhermeleobas/260/orig -> origin/gh/guilhermeleobas/260/orig 2025-12-04T09:20:54.4065365Z * [new branch] gh/guilhermeleobas/261/base -> origin/gh/guilhermeleobas/261/base 2025-12-04T09:20:54.4065520Z * [new branch] gh/guilhermeleobas/261/head -> origin/gh/guilhermeleobas/261/head 2025-12-04T09:20:54.4065670Z * [new branch] gh/guilhermeleobas/261/orig -> origin/gh/guilhermeleobas/261/orig 2025-12-04T09:20:54.4065976Z * [new branch] gh/guilhermeleobas/262/base -> origin/gh/guilhermeleobas/262/base 2025-12-04T09:20:54.4066124Z * [new branch] gh/guilhermeleobas/262/head -> origin/gh/guilhermeleobas/262/head 2025-12-04T09:20:54.4066272Z * [new branch] gh/guilhermeleobas/262/orig -> origin/gh/guilhermeleobas/262/orig 2025-12-04T09:20:54.4066473Z * [new branch] gh/guilhermeleobas/263/base -> origin/gh/guilhermeleobas/263/base 2025-12-04T09:20:54.4066626Z * [new branch] gh/guilhermeleobas/263/head -> origin/gh/guilhermeleobas/263/head 2025-12-04T09:20:54.4066782Z * [new branch] gh/guilhermeleobas/263/orig -> origin/gh/guilhermeleobas/263/orig 2025-12-04T09:20:54.4066929Z * [new branch] gh/guilhermeleobas/264/base -> origin/gh/guilhermeleobas/264/base 2025-12-04T09:20:54.4067075Z * [new branch] gh/guilhermeleobas/264/head -> origin/gh/guilhermeleobas/264/head 2025-12-04T09:20:54.4067230Z * [new branch] gh/guilhermeleobas/264/orig -> origin/gh/guilhermeleobas/264/orig 2025-12-04T09:20:54.4067375Z * [new branch] gh/guilhermeleobas/265/base -> origin/gh/guilhermeleobas/265/base 2025-12-04T09:20:54.4067534Z * [new branch] gh/guilhermeleobas/265/head -> origin/gh/guilhermeleobas/265/head 2025-12-04T09:20:54.4067680Z * [new branch] gh/guilhermeleobas/265/orig -> origin/gh/guilhermeleobas/265/orig 2025-12-04T09:20:54.4067949Z * [new branch] gh/guilhermeleobas/266/base -> origin/gh/guilhermeleobas/266/base 2025-12-04T09:20:54.4073117Z * [new branch] gh/guilhermeleobas/266/head -> origin/gh/guilhermeleobas/266/head 2025-12-04T09:20:54.4077255Z * [new branch] gh/guilhermeleobas/266/orig -> origin/gh/guilhermeleobas/266/orig 2025-12-04T09:20:54.4082772Z * [new branch] gh/guilhermeleobas/267/base -> origin/gh/guilhermeleobas/267/base 2025-12-04T09:20:54.4085014Z * [new branch] gh/guilhermeleobas/267/head -> origin/gh/guilhermeleobas/267/head 2025-12-04T09:20:54.4085393Z * [new branch] gh/guilhermeleobas/267/orig -> origin/gh/guilhermeleobas/267/orig 2025-12-04T09:20:54.4085643Z * [new branch] gh/hameerabbasi/1/base -> origin/gh/hameerabbasi/1/base 2025-12-04T09:20:54.4085833Z * [new branch] gh/hameerabbasi/1/head -> origin/gh/hameerabbasi/1/head 2025-12-04T09:20:54.4085968Z * [new branch] gh/hameerabbasi/2/base -> origin/gh/hameerabbasi/2/base 2025-12-04T09:20:54.4086102Z * [new branch] gh/hameerabbasi/2/head -> origin/gh/hameerabbasi/2/head 2025-12-04T09:20:54.4086246Z * [new branch] gh/hameerabbasi/2/orig -> origin/gh/hameerabbasi/2/orig 2025-12-04T09:20:54.4086381Z * [new branch] gh/hameerabbasi/3/base -> origin/gh/hameerabbasi/3/base 2025-12-04T09:20:54.4086523Z * [new branch] gh/hameerabbasi/3/head -> origin/gh/hameerabbasi/3/head 2025-12-04T09:20:54.4086664Z * [new branch] gh/hameerabbasi/3/orig -> origin/gh/hameerabbasi/3/orig 2025-12-04T09:20:54.4086796Z * [new branch] gh/hameerabbasi/4/base -> origin/gh/hameerabbasi/4/base 2025-12-04T09:20:54.4086936Z * [new branch] gh/hameerabbasi/4/head -> origin/gh/hameerabbasi/4/head 2025-12-04T09:20:54.4087068Z * [new branch] gh/hameerabbasi/4/orig -> origin/gh/hameerabbasi/4/orig 2025-12-04T09:20:54.4087197Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-12-04T09:20:54.4087327Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-12-04T09:20:54.4087460Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-12-04T09:20:54.4087585Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-12-04T09:20:54.4087704Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-12-04T09:20:54.4088033Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-12-04T09:20:54.4088178Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-12-04T09:20:54.4088305Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-12-04T09:20:54.4088508Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-12-04T09:20:54.4088640Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-12-04T09:20:54.4088772Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-12-04T09:20:54.4088906Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-12-04T09:20:54.4089032Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-12-04T09:20:54.4090454Z * [new branch] gh/isuruf/158/base -> origin/gh/isuruf/158/base 2025-12-04T09:20:54.4090605Z * [new branch] gh/isuruf/158/head -> origin/gh/isuruf/158/head 2025-12-04T09:20:54.4092538Z * [new branch] gh/isuruf/159/base -> origin/gh/isuruf/159/base 2025-12-04T09:20:54.4092876Z * [new branch] gh/isuruf/159/head -> origin/gh/isuruf/159/head 2025-12-04T09:20:54.4093077Z * [new branch] gh/isuruf/160/base -> origin/gh/isuruf/160/base 2025-12-04T09:20:54.4095802Z * [new branch] gh/isuruf/160/head -> origin/gh/isuruf/160/head 2025-12-04T09:20:54.4096130Z * [new branch] gh/isuruf/160/orig -> origin/gh/isuruf/160/orig 2025-12-04T09:20:54.4096320Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-12-04T09:20:54.4096466Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-12-04T09:20:54.4096682Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-12-04T09:20:54.4102842Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-12-04T09:20:54.4104996Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-12-04T09:20:54.4105276Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-12-04T09:20:54.4111991Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-12-04T09:20:54.4116951Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-12-04T09:20:54.4121094Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-12-04T09:20:54.4125340Z * [new branch] gh/jamesjwu/196/base -> origin/gh/jamesjwu/196/base 2025-12-04T09:20:54.4127350Z * [new branch] gh/jamesjwu/196/head -> origin/gh/jamesjwu/196/head 2025-12-04T09:20:54.4127509Z * [new branch] gh/jamesjwu/196/orig -> origin/gh/jamesjwu/196/orig 2025-12-04T09:20:54.4127683Z * [new branch] gh/jamesjwu/198/base -> origin/gh/jamesjwu/198/base 2025-12-04T09:20:54.4127824Z * [new branch] gh/jamesjwu/198/head -> origin/gh/jamesjwu/198/head 2025-12-04T09:20:54.4127974Z * [new branch] gh/jamesjwu/198/orig -> origin/gh/jamesjwu/198/orig 2025-12-04T09:20:54.4128121Z * [new branch] gh/jamesjwu/207/base -> origin/gh/jamesjwu/207/base 2025-12-04T09:20:54.4128268Z * [new branch] gh/jamesjwu/207/head -> origin/gh/jamesjwu/207/head 2025-12-04T09:20:54.4128404Z * [new branch] gh/jamesjwu/207/orig -> origin/gh/jamesjwu/207/orig 2025-12-04T09:20:54.4128539Z * [new branch] gh/jamesjwu/208/base -> origin/gh/jamesjwu/208/base 2025-12-04T09:20:54.4128684Z * [new branch] gh/jamesjwu/208/head -> origin/gh/jamesjwu/208/head 2025-12-04T09:20:54.4129073Z * [new branch] gh/jamesjwu/208/orig -> origin/gh/jamesjwu/208/orig 2025-12-04T09:20:54.4129231Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-12-04T09:20:54.4129374Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-12-04T09:20:54.4129555Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-12-04T09:20:54.4129716Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-12-04T09:20:54.4129854Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-12-04T09:20:54.4129999Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-12-04T09:20:54.4130134Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-12-04T09:20:54.4130272Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-12-04T09:20:54.4130426Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-12-04T09:20:54.4130566Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-12-04T09:20:54.4130719Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-12-04T09:20:54.4130861Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-12-04T09:20:54.4130995Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-12-04T09:20:54.4131137Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-12-04T09:20:54.4131274Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-12-04T09:20:54.4131417Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-12-04T09:20:54.4131700Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-12-04T09:20:54.4131854Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-12-04T09:20:54.4131996Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-12-04T09:20:54.4132152Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-12-04T09:20:54.4132290Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-12-04T09:20:54.4132445Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-12-04T09:20:54.4132588Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-12-04T09:20:54.4132737Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-12-04T09:20:54.4132882Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-12-04T09:20:54.4133025Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-12-04T09:20:54.4133185Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-12-04T09:20:54.4137645Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-12-04T09:20:54.4142709Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-12-04T09:20:54.4146090Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-12-04T09:20:54.4150171Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-12-04T09:20:54.4154331Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-12-04T09:20:54.4154824Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-12-04T09:20:54.4154992Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-12-04T09:20:54.4155137Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-12-04T09:20:54.4155372Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-12-04T09:20:54.4155504Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-12-04T09:20:54.4155628Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-12-04T09:20:54.4155792Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-12-04T09:20:54.4155927Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-12-04T09:20:54.4156051Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-12-04T09:20:54.4156184Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-12-04T09:20:54.4156306Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-12-04T09:20:54.4156463Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-12-04T09:20:54.4156599Z * [new branch] gh/janeyx99/305/base -> origin/gh/janeyx99/305/base 2025-12-04T09:20:54.4156723Z * [new branch] gh/janeyx99/305/head -> origin/gh/janeyx99/305/head 2025-12-04T09:20:54.4156857Z * [new branch] gh/janeyx99/306/base -> origin/gh/janeyx99/306/base 2025-12-04T09:20:54.4156981Z * [new branch] gh/janeyx99/306/head -> origin/gh/janeyx99/306/head 2025-12-04T09:20:54.4157102Z * [new branch] gh/janeyx99/314/base -> origin/gh/janeyx99/314/base 2025-12-04T09:20:54.4157242Z * [new branch] gh/janeyx99/314/head -> origin/gh/janeyx99/314/head 2025-12-04T09:20:54.4157362Z * [new branch] gh/janeyx99/314/orig -> origin/gh/janeyx99/314/orig 2025-12-04T09:20:54.4157482Z * [new branch] gh/janeyx99/315/base -> origin/gh/janeyx99/315/base 2025-12-04T09:20:54.4157614Z * [new branch] gh/janeyx99/315/head -> origin/gh/janeyx99/315/head 2025-12-04T09:20:54.4157734Z * [new branch] gh/janeyx99/315/orig -> origin/gh/janeyx99/315/orig 2025-12-04T09:20:54.4157862Z * [new branch] gh/janeyx99/316/base -> origin/gh/janeyx99/316/base 2025-12-04T09:20:54.4157988Z * [new branch] gh/janeyx99/316/head -> origin/gh/janeyx99/316/head 2025-12-04T09:20:54.4158111Z * [new branch] gh/janeyx99/316/orig -> origin/gh/janeyx99/316/orig 2025-12-04T09:20:54.4158396Z * [new branch] gh/janeyx99/317/base -> origin/gh/janeyx99/317/base 2025-12-04T09:20:54.4158602Z * [new branch] gh/janeyx99/317/head -> origin/gh/janeyx99/317/head 2025-12-04T09:20:54.4159231Z * [new branch] gh/janeyx99/317/orig -> origin/gh/janeyx99/317/orig 2025-12-04T09:20:54.4159571Z * [new branch] gh/janeyx99/325/base -> origin/gh/janeyx99/325/base 2025-12-04T09:20:54.4159737Z * [new branch] gh/janeyx99/325/head -> origin/gh/janeyx99/325/head 2025-12-04T09:20:54.4161031Z * [new branch] gh/janeyx99/325/orig -> origin/gh/janeyx99/325/orig 2025-12-04T09:20:54.4161424Z * [new branch] gh/janeyx99/327/base -> origin/gh/janeyx99/327/base 2025-12-04T09:20:54.4164462Z * [new branch] gh/janeyx99/327/head -> origin/gh/janeyx99/327/head 2025-12-04T09:20:54.4164791Z * [new branch] gh/janeyx99/327/orig -> origin/gh/janeyx99/327/orig 2025-12-04T09:20:54.4164975Z * [new branch] gh/janeyx99/328/base -> origin/gh/janeyx99/328/base 2025-12-04T09:20:54.4165158Z * [new branch] gh/janeyx99/328/head -> origin/gh/janeyx99/328/head 2025-12-04T09:20:54.4165300Z * [new branch] gh/janeyx99/328/orig -> origin/gh/janeyx99/328/orig 2025-12-04T09:20:54.4166514Z * [new branch] gh/janeyx99/329/base -> origin/gh/janeyx99/329/base 2025-12-04T09:20:54.4166915Z * [new branch] gh/janeyx99/329/head -> origin/gh/janeyx99/329/head 2025-12-04T09:20:54.4169457Z * [new branch] gh/janeyx99/329/orig -> origin/gh/janeyx99/329/orig 2025-12-04T09:20:54.4169630Z * [new branch] gh/janeyx99/330/base -> origin/gh/janeyx99/330/base 2025-12-04T09:20:54.4169956Z * [new branch] gh/janeyx99/330/head -> origin/gh/janeyx99/330/head 2025-12-04T09:20:54.4170138Z * [new branch] gh/janeyx99/330/orig -> origin/gh/janeyx99/330/orig 2025-12-04T09:20:54.4171899Z * [new branch] gh/janeyx99/331/base -> origin/gh/janeyx99/331/base 2025-12-04T09:20:54.4172083Z * [new branch] gh/janeyx99/331/head -> origin/gh/janeyx99/331/head 2025-12-04T09:20:54.4172887Z * [new branch] gh/janeyx99/331/orig -> origin/gh/janeyx99/331/orig 2025-12-04T09:20:54.4177715Z * [new branch] gh/janeyx99/332/base -> origin/gh/janeyx99/332/base 2025-12-04T09:20:54.4183596Z * [new branch] gh/janeyx99/332/head -> origin/gh/janeyx99/332/head 2025-12-04T09:20:54.4183767Z * [new branch] gh/janeyx99/332/orig -> origin/gh/janeyx99/332/orig 2025-12-04T09:20:54.4183924Z * [new branch] gh/janeyx99/333/base -> origin/gh/janeyx99/333/base 2025-12-04T09:20:54.4184059Z * [new branch] gh/janeyx99/333/head -> origin/gh/janeyx99/333/head 2025-12-04T09:20:54.4184182Z * [new branch] gh/janeyx99/333/orig -> origin/gh/janeyx99/333/orig 2025-12-04T09:20:54.4184325Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-12-04T09:20:54.4184454Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-12-04T09:20:54.4184578Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-12-04T09:20:54.4189752Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-12-04T09:20:54.4190104Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-12-04T09:20:54.4190302Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-12-04T09:20:54.4190451Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-12-04T09:20:54.4190591Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-12-04T09:20:54.4190753Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-12-04T09:20:54.4190892Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-12-04T09:20:54.4192298Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-12-04T09:20:54.4192467Z * [new branch] gh/jansel/533/base -> origin/gh/jansel/533/base 2025-12-04T09:20:54.4192618Z * [new branch] gh/jansel/533/head -> origin/gh/jansel/533/head 2025-12-04T09:20:54.4192749Z * [new branch] gh/jansel/533/orig -> origin/gh/jansel/533/orig 2025-12-04T09:20:54.4192882Z * [new branch] gh/jansel/552/base -> origin/gh/jansel/552/base 2025-12-04T09:20:54.4193037Z * [new branch] gh/jansel/552/head -> origin/gh/jansel/552/head 2025-12-04T09:20:54.4198306Z * [new branch] gh/jansel/552/orig -> origin/gh/jansel/552/orig 2025-12-04T09:20:54.4198480Z * [new branch] gh/jansel/553/base -> origin/gh/jansel/553/base 2025-12-04T09:20:54.4198618Z * [new branch] gh/jansel/553/head -> origin/gh/jansel/553/head 2025-12-04T09:20:54.4198743Z * [new branch] gh/jansel/553/orig -> origin/gh/jansel/553/orig 2025-12-04T09:20:54.4198871Z * [new branch] gh/jansel/554/base -> origin/gh/jansel/554/base 2025-12-04T09:20:54.4199433Z * [new branch] gh/jansel/554/head -> origin/gh/jansel/554/head 2025-12-04T09:20:54.4199561Z * [new branch] gh/jansel/554/orig -> origin/gh/jansel/554/orig 2025-12-04T09:20:54.4203216Z * [new branch] gh/jansel/555/base -> origin/gh/jansel/555/base 2025-12-04T09:20:54.4203638Z * [new branch] gh/jansel/555/head -> origin/gh/jansel/555/head 2025-12-04T09:20:54.4203793Z * [new branch] gh/jansel/555/orig -> origin/gh/jansel/555/orig 2025-12-04T09:20:54.4203952Z * [new branch] gh/jansel/556/base -> origin/gh/jansel/556/base 2025-12-04T09:20:54.4204100Z * [new branch] gh/jansel/556/head -> origin/gh/jansel/556/head 2025-12-04T09:20:54.4204254Z * [new branch] gh/jansel/556/orig -> origin/gh/jansel/556/orig 2025-12-04T09:20:54.4204399Z * [new branch] gh/jansel/557/base -> origin/gh/jansel/557/base 2025-12-04T09:20:54.4204968Z * [new branch] gh/jansel/557/head -> origin/gh/jansel/557/head 2025-12-04T09:20:54.4205127Z * [new branch] gh/jansel/557/orig -> origin/gh/jansel/557/orig 2025-12-04T09:20:54.4205267Z * [new branch] gh/jansel/558/base -> origin/gh/jansel/558/base 2025-12-04T09:20:54.4205418Z * [new branch] gh/jansel/558/head -> origin/gh/jansel/558/head 2025-12-04T09:20:54.4205557Z * [new branch] gh/jansel/558/orig -> origin/gh/jansel/558/orig 2025-12-04T09:20:54.4205700Z * [new branch] gh/jansel/559/base -> origin/gh/jansel/559/base 2025-12-04T09:20:54.4208625Z * [new branch] gh/jansel/559/head -> origin/gh/jansel/559/head 2025-12-04T09:20:54.4212862Z * [new branch] gh/jansel/559/orig -> origin/gh/jansel/559/orig 2025-12-04T09:20:54.4213057Z * [new branch] gh/jansel/560/base -> origin/gh/jansel/560/base 2025-12-04T09:20:54.4213214Z * [new branch] gh/jansel/560/head -> origin/gh/jansel/560/head 2025-12-04T09:20:54.4213349Z * [new branch] gh/jansel/560/orig -> origin/gh/jansel/560/orig 2025-12-04T09:20:54.4213500Z * [new branch] gh/jansel/561/base -> origin/gh/jansel/561/base 2025-12-04T09:20:54.4213651Z * [new branch] gh/jansel/561/head -> origin/gh/jansel/561/head 2025-12-04T09:20:54.4213788Z * [new branch] gh/jansel/561/orig -> origin/gh/jansel/561/orig 2025-12-04T09:20:54.4218759Z * [new branch] gh/jansel/562/base -> origin/gh/jansel/562/base 2025-12-04T09:20:54.4218960Z * [new branch] gh/jansel/562/head -> origin/gh/jansel/562/head 2025-12-04T09:20:54.4219128Z * [new branch] gh/jansel/562/orig -> origin/gh/jansel/562/orig 2025-12-04T09:20:54.4219392Z * [new branch] gh/jansel/563/base -> origin/gh/jansel/563/base 2025-12-04T09:20:54.4219905Z * [new branch] gh/jansel/563/head -> origin/gh/jansel/563/head 2025-12-04T09:20:54.4220078Z * [new branch] gh/jansel/563/orig -> origin/gh/jansel/563/orig 2025-12-04T09:20:54.4220217Z * [new branch] gh/jansel/564/base -> origin/gh/jansel/564/base 2025-12-04T09:20:54.4220935Z * [new branch] gh/jansel/564/head -> origin/gh/jansel/564/head 2025-12-04T09:20:54.4221082Z * [new branch] gh/jansel/564/orig -> origin/gh/jansel/564/orig 2025-12-04T09:20:54.4221230Z * [new branch] gh/jansel/565/base -> origin/gh/jansel/565/base 2025-12-04T09:20:54.4221377Z * [new branch] gh/jansel/565/head -> origin/gh/jansel/565/head 2025-12-04T09:20:54.4221516Z * [new branch] gh/jansel/565/orig -> origin/gh/jansel/565/orig 2025-12-04T09:20:54.4221663Z * [new branch] gh/jansel/566/base -> origin/gh/jansel/566/base 2025-12-04T09:20:54.4222015Z * [new branch] gh/jansel/566/head -> origin/gh/jansel/566/head 2025-12-04T09:20:54.4222161Z * [new branch] gh/jansel/566/orig -> origin/gh/jansel/566/orig 2025-12-04T09:20:54.4222612Z * [new branch] gh/jansel/567/base -> origin/gh/jansel/567/base 2025-12-04T09:20:54.4228587Z * [new branch] gh/jansel/567/head -> origin/gh/jansel/567/head 2025-12-04T09:20:54.4228952Z * [new branch] gh/jansel/567/orig -> origin/gh/jansel/567/orig 2025-12-04T09:20:54.4229119Z * [new branch] gh/jansel/568/base -> origin/gh/jansel/568/base 2025-12-04T09:20:54.4229260Z * [new branch] gh/jansel/568/head -> origin/gh/jansel/568/head 2025-12-04T09:20:54.4229550Z * [new branch] gh/jansel/568/orig -> origin/gh/jansel/568/orig 2025-12-04T09:20:54.4229757Z * [new branch] gh/jansel/569/base -> origin/gh/jansel/569/base 2025-12-04T09:20:54.4234153Z * [new branch] gh/jansel/569/head -> origin/gh/jansel/569/head 2025-12-04T09:20:54.4234484Z * [new branch] gh/jansel/569/orig -> origin/gh/jansel/569/orig 2025-12-04T09:20:54.4234755Z * [new branch] gh/jansel/570/base -> origin/gh/jansel/570/base 2025-12-04T09:20:54.4235042Z * [new branch] gh/jansel/570/head -> origin/gh/jansel/570/head 2025-12-04T09:20:54.4235252Z * [new branch] gh/jansel/570/orig -> origin/gh/jansel/570/orig 2025-12-04T09:20:54.4235393Z * [new branch] gh/jansel/571/base -> origin/gh/jansel/571/base 2025-12-04T09:20:54.4235532Z * [new branch] gh/jansel/571/head -> origin/gh/jansel/571/head 2025-12-04T09:20:54.4235662Z * [new branch] gh/jansel/571/orig -> origin/gh/jansel/571/orig 2025-12-04T09:20:54.4235804Z * [new branch] gh/jansel/572/base -> origin/gh/jansel/572/base 2025-12-04T09:20:54.4235927Z * [new branch] gh/jansel/572/head -> origin/gh/jansel/572/head 2025-12-04T09:20:54.4236060Z * [new branch] gh/jansel/572/orig -> origin/gh/jansel/572/orig 2025-12-04T09:20:54.4236321Z * [new branch] gh/jansel/573/base -> origin/gh/jansel/573/base 2025-12-04T09:20:54.4236543Z * [new branch] gh/jansel/573/head -> origin/gh/jansel/573/head 2025-12-04T09:20:54.4240974Z * [new branch] gh/jansel/573/orig -> origin/gh/jansel/573/orig 2025-12-04T09:20:54.4241334Z * [new branch] gh/jansel/574/base -> origin/gh/jansel/574/base 2025-12-04T09:20:54.4241517Z * [new branch] gh/jansel/574/head -> origin/gh/jansel/574/head 2025-12-04T09:20:54.4241663Z * [new branch] gh/jansel/574/orig -> origin/gh/jansel/574/orig 2025-12-04T09:20:54.4241831Z * [new branch] gh/jansel/575/base -> origin/gh/jansel/575/base 2025-12-04T09:20:54.4242133Z * [new branch] gh/jansel/575/head -> origin/gh/jansel/575/head 2025-12-04T09:20:54.4242580Z * [new branch] gh/jansel/575/orig -> origin/gh/jansel/575/orig 2025-12-04T09:20:54.4245243Z * [new branch] gh/jansel/576/base -> origin/gh/jansel/576/base 2025-12-04T09:20:54.4245553Z * [new branch] gh/jansel/576/head -> origin/gh/jansel/576/head 2025-12-04T09:20:54.4245868Z * [new branch] gh/jansel/576/orig -> origin/gh/jansel/576/orig 2025-12-04T09:20:54.4246043Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-12-04T09:20:54.4246215Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-12-04T09:20:54.4246795Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-12-04T09:20:54.4249787Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-12-04T09:20:54.4249995Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-12-04T09:20:54.4250159Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-12-04T09:20:54.4250496Z * [new branch] gh/jerryzh168/1/base -> origin/gh/jerryzh168/1/base 2025-12-04T09:20:54.4250657Z * [new branch] gh/jerryzh168/1/head -> origin/gh/jerryzh168/1/head 2025-12-04T09:20:54.4255946Z * [new branch] gh/jerryzh168/1/orig -> origin/gh/jerryzh168/1/orig 2025-12-04T09:20:54.4256122Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-12-04T09:20:54.4256259Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-12-04T09:20:54.4256402Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-12-04T09:20:54.4256553Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-12-04T09:20:54.4258529Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-12-04T09:20:54.4258726Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-12-04T09:20:54.4258908Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-12-04T09:20:54.4259056Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-12-04T09:20:54.4259215Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-12-04T09:20:54.4259363Z * [new branch] gh/jiayisunx/77/base -> origin/gh/jiayisunx/77/base 2025-12-04T09:20:54.4259687Z * [new branch] gh/jiayisunx/77/head -> origin/gh/jiayisunx/77/head 2025-12-04T09:20:54.4265002Z * [new branch] gh/jiayisunx/77/orig -> origin/gh/jiayisunx/77/orig 2025-12-04T09:20:54.4265167Z * [new branch] gh/jiayisunx/78/base -> origin/gh/jiayisunx/78/base 2025-12-04T09:20:54.4265543Z * [new branch] gh/jiayisunx/78/head -> origin/gh/jiayisunx/78/head 2025-12-04T09:20:54.4265699Z * [new branch] gh/jiayisunx/78/orig -> origin/gh/jiayisunx/78/orig 2025-12-04T09:20:54.4265832Z * [new branch] gh/jiayisunx/79/base -> origin/gh/jiayisunx/79/base 2025-12-04T09:20:54.4265958Z * [new branch] gh/jiayisunx/79/head -> origin/gh/jiayisunx/79/head 2025-12-04T09:20:54.4266084Z * [new branch] gh/jiayisunx/79/orig -> origin/gh/jiayisunx/79/orig 2025-12-04T09:20:54.4268338Z * [new branch] gh/jiayisunx/82/base -> origin/gh/jiayisunx/82/base 2025-12-04T09:20:54.4268532Z * [new branch] gh/jiayisunx/82/head -> origin/gh/jiayisunx/82/head 2025-12-04T09:20:54.4268709Z * [new branch] gh/jiayisunx/82/orig -> origin/gh/jiayisunx/82/orig 2025-12-04T09:20:54.4268862Z * [new branch] gh/jiayisunx/83/base -> origin/gh/jiayisunx/83/base 2025-12-04T09:20:54.4269011Z * [new branch] gh/jiayisunx/83/head -> origin/gh/jiayisunx/83/head 2025-12-04T09:20:54.4269188Z * [new branch] gh/jiayisunx/83/orig -> origin/gh/jiayisunx/83/orig 2025-12-04T09:20:54.4269334Z * [new branch] gh/jiayisunx/84/base -> origin/gh/jiayisunx/84/base 2025-12-04T09:20:54.4274115Z * [new branch] gh/jiayisunx/84/head -> origin/gh/jiayisunx/84/head 2025-12-04T09:20:54.4274436Z * [new branch] gh/jiayisunx/84/orig -> origin/gh/jiayisunx/84/orig 2025-12-04T09:20:54.4274596Z * [new branch] gh/jiayisunx/85/base -> origin/gh/jiayisunx/85/base 2025-12-04T09:20:54.4274736Z * [new branch] gh/jiayisunx/85/head -> origin/gh/jiayisunx/85/head 2025-12-04T09:20:54.4275067Z * [new branch] gh/jiayisunx/85/orig -> origin/gh/jiayisunx/85/orig 2025-12-04T09:20:54.4275362Z * [new branch] gh/jiayisunx/86/base -> origin/gh/jiayisunx/86/base 2025-12-04T09:20:54.4275523Z * [new branch] gh/jiayisunx/86/head -> origin/gh/jiayisunx/86/head 2025-12-04T09:20:54.4276092Z * [new branch] gh/jiayisunx/86/orig -> origin/gh/jiayisunx/86/orig 2025-12-04T09:20:54.4276226Z * [new branch] gh/jiayisunx/87/base -> origin/gh/jiayisunx/87/base 2025-12-04T09:20:54.4276355Z * [new branch] gh/jiayisunx/87/head -> origin/gh/jiayisunx/87/head 2025-12-04T09:20:54.4276486Z * [new branch] gh/jiayisunx/87/orig -> origin/gh/jiayisunx/87/orig 2025-12-04T09:20:54.4276618Z * [new branch] gh/jiayisunx/88/base -> origin/gh/jiayisunx/88/base 2025-12-04T09:20:54.4278190Z * [new branch] gh/jiayisunx/88/head -> origin/gh/jiayisunx/88/head 2025-12-04T09:20:54.4278534Z * [new branch] gh/jiayisunx/88/orig -> origin/gh/jiayisunx/88/orig 2025-12-04T09:20:54.4284129Z * [new branch] gh/jiayisunx/89/base -> origin/gh/jiayisunx/89/base 2025-12-04T09:20:54.4284448Z * [new branch] gh/jiayisunx/89/head -> origin/gh/jiayisunx/89/head 2025-12-04T09:20:54.4284819Z * [new branch] gh/jiayisunx/89/orig -> origin/gh/jiayisunx/89/orig 2025-12-04T09:20:54.4285026Z * [new branch] gh/jiayisunx/90/base -> origin/gh/jiayisunx/90/base 2025-12-04T09:20:54.4285156Z * [new branch] gh/jiayisunx/90/head -> origin/gh/jiayisunx/90/head 2025-12-04T09:20:54.4285426Z * [new branch] gh/jiayisunx/90/orig -> origin/gh/jiayisunx/90/orig 2025-12-04T09:20:54.4285978Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-12-04T09:20:54.4286179Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-12-04T09:20:54.4286335Z * [new branch] gh/jturney/1/base -> origin/gh/jturney/1/base 2025-12-04T09:20:54.4286494Z * [new branch] gh/jturney/1/head -> origin/gh/jturney/1/head 2025-12-04T09:20:54.4286662Z * [new branch] gh/jturney/1/orig -> origin/gh/jturney/1/orig 2025-12-04T09:20:54.4287050Z * [new branch] gh/jturney/2/base -> origin/gh/jturney/2/base 2025-12-04T09:20:54.4287238Z * [new branch] gh/jturney/2/head -> origin/gh/jturney/2/head 2025-12-04T09:20:54.4287388Z * [new branch] gh/jturney/2/orig -> origin/gh/jturney/2/orig 2025-12-04T09:20:54.4291742Z * [new branch] gh/karthickai/10/base -> origin/gh/karthickai/10/base 2025-12-04T09:20:54.4291972Z * [new branch] gh/karthickai/10/head -> origin/gh/karthickai/10/head 2025-12-04T09:20:54.4292154Z * [new branch] gh/karthickai/10/orig -> origin/gh/karthickai/10/orig 2025-12-04T09:20:54.4292309Z * [new branch] gh/karthickai/11/base -> origin/gh/karthickai/11/base 2025-12-04T09:20:54.4292482Z * [new branch] gh/karthickai/11/head -> origin/gh/karthickai/11/head 2025-12-04T09:20:54.4298155Z * [new branch] gh/karthickai/11/orig -> origin/gh/karthickai/11/orig 2025-12-04T09:20:54.4298486Z * [new branch] gh/karthickai/12/base -> origin/gh/karthickai/12/base 2025-12-04T09:20:54.4298833Z * [new branch] gh/karthickai/12/head -> origin/gh/karthickai/12/head 2025-12-04T09:20:54.4299085Z * [new branch] gh/karthickai/12/orig -> origin/gh/karthickai/12/orig 2025-12-04T09:20:54.4299238Z * [new branch] gh/karthickai/13/base -> origin/gh/karthickai/13/base 2025-12-04T09:20:54.4299396Z * [new branch] gh/karthickai/13/head -> origin/gh/karthickai/13/head 2025-12-04T09:20:54.4304122Z * [new branch] gh/karthickai/13/orig -> origin/gh/karthickai/13/orig 2025-12-04T09:20:54.4304451Z * [new branch] gh/karthickai/14/base -> origin/gh/karthickai/14/base 2025-12-04T09:20:54.4304661Z * [new branch] gh/karthickai/14/head -> origin/gh/karthickai/14/head 2025-12-04T09:20:54.4304950Z * [new branch] gh/karthickai/14/orig -> origin/gh/karthickai/14/orig 2025-12-04T09:20:54.4305228Z * [new branch] gh/karthickai/15/base -> origin/gh/karthickai/15/base 2025-12-04T09:20:54.4305424Z * [new branch] gh/karthickai/15/head -> origin/gh/karthickai/15/head 2025-12-04T09:20:54.4305557Z * [new branch] gh/karthickai/15/orig -> origin/gh/karthickai/15/orig 2025-12-04T09:20:54.4305696Z * [new branch] gh/karthickai/16/base -> origin/gh/karthickai/16/base 2025-12-04T09:20:54.4306176Z * [new branch] gh/karthickai/16/head -> origin/gh/karthickai/16/head 2025-12-04T09:20:54.4306359Z * [new branch] gh/karthickai/16/orig -> origin/gh/karthickai/16/orig 2025-12-04T09:20:54.4311152Z * [new branch] gh/karthickai/17/base -> origin/gh/karthickai/17/base 2025-12-04T09:20:54.4311481Z * [new branch] gh/karthickai/17/head -> origin/gh/karthickai/17/head 2025-12-04T09:20:54.4311656Z * [new branch] gh/karthickai/17/orig -> origin/gh/karthickai/17/orig 2025-12-04T09:20:54.4311799Z * [new branch] gh/karthickai/18/base -> origin/gh/karthickai/18/base 2025-12-04T09:20:54.4311930Z * [new branch] gh/karthickai/18/head -> origin/gh/karthickai/18/head 2025-12-04T09:20:54.4312061Z * [new branch] gh/karthickai/18/orig -> origin/gh/karthickai/18/orig 2025-12-04T09:20:54.4312338Z * [new branch] gh/karthickai/19/base -> origin/gh/karthickai/19/base 2025-12-04T09:20:54.4317098Z * [new branch] gh/karthickai/19/head -> origin/gh/karthickai/19/head 2025-12-04T09:20:54.4317426Z * [new branch] gh/karthickai/19/orig -> origin/gh/karthickai/19/orig 2025-12-04T09:20:54.4317688Z * [new branch] gh/karthickai/20/base -> origin/gh/karthickai/20/base 2025-12-04T09:20:54.4317934Z * [new branch] gh/karthickai/20/head -> origin/gh/karthickai/20/head 2025-12-04T09:20:54.4318197Z * [new branch] gh/karthickai/20/orig -> origin/gh/karthickai/20/orig 2025-12-04T09:20:54.4318333Z * [new branch] gh/karthickai/21/base -> origin/gh/karthickai/21/base 2025-12-04T09:20:54.4318477Z * [new branch] gh/karthickai/21/head -> origin/gh/karthickai/21/head 2025-12-04T09:20:54.4318609Z * [new branch] gh/karthickai/21/orig -> origin/gh/karthickai/21/orig 2025-12-04T09:20:54.4322227Z * [new branch] gh/karthickai/22/base -> origin/gh/karthickai/22/base 2025-12-04T09:20:54.4322401Z * [new branch] gh/karthickai/22/head -> origin/gh/karthickai/22/head 2025-12-04T09:20:54.4322569Z * [new branch] gh/karthickai/22/orig -> origin/gh/karthickai/22/orig 2025-12-04T09:20:54.4322717Z * [new branch] gh/karthickai/23/base -> origin/gh/karthickai/23/base 2025-12-04T09:20:54.4322862Z * [new branch] gh/karthickai/23/head -> origin/gh/karthickai/23/head 2025-12-04T09:20:54.4323007Z * [new branch] gh/karthickai/23/orig -> origin/gh/karthickai/23/orig 2025-12-04T09:20:54.4323143Z * [new branch] gh/karthickai/24/base -> origin/gh/karthickai/24/base 2025-12-04T09:20:54.4323283Z * [new branch] gh/karthickai/24/head -> origin/gh/karthickai/24/head 2025-12-04T09:20:54.4325555Z * [new branch] gh/karthickai/24/orig -> origin/gh/karthickai/24/orig 2025-12-04T09:20:54.4325817Z * [new branch] gh/karthickai/25/base -> origin/gh/karthickai/25/base 2025-12-04T09:20:54.4326105Z * [new branch] gh/karthickai/25/head -> origin/gh/karthickai/25/head 2025-12-04T09:20:54.4326256Z * [new branch] gh/karthickai/25/orig -> origin/gh/karthickai/25/orig 2025-12-04T09:20:54.4326397Z * [new branch] gh/karthickai/26/base -> origin/gh/karthickai/26/base 2025-12-04T09:20:54.4327348Z * [new branch] gh/karthickai/26/head -> origin/gh/karthickai/26/head 2025-12-04T09:20:54.4327528Z * [new branch] gh/karthickai/26/orig -> origin/gh/karthickai/26/orig 2025-12-04T09:20:54.4329966Z * [new branch] gh/karthickai/6/base -> origin/gh/karthickai/6/base 2025-12-04T09:20:54.4331105Z * [new branch] gh/karthickai/6/head -> origin/gh/karthickai/6/head 2025-12-04T09:20:54.4331537Z * [new branch] gh/karthickai/6/orig -> origin/gh/karthickai/6/orig 2025-12-04T09:20:54.4333435Z * [new branch] gh/krocki/1/base -> origin/gh/krocki/1/base 2025-12-04T09:20:54.4333809Z * [new branch] gh/krocki/1/head -> origin/gh/krocki/1/head 2025-12-04T09:20:54.4335005Z * [new branch] gh/krocki/1/orig -> origin/gh/krocki/1/orig 2025-12-04T09:20:54.4335979Z * [new branch] gh/krocki/2/base -> origin/gh/krocki/2/base 2025-12-04T09:20:54.4336651Z * [new branch] gh/krocki/2/head -> origin/gh/krocki/2/head 2025-12-04T09:20:54.4336926Z * [new branch] gh/krocki/2/orig -> origin/gh/krocki/2/orig 2025-12-04T09:20:54.4341765Z * [new branch] gh/kurtamohler/60/base -> origin/gh/kurtamohler/60/base 2025-12-04T09:20:54.4341958Z * [new branch] gh/kurtamohler/60/head -> origin/gh/kurtamohler/60/head 2025-12-04T09:20:54.4342103Z * [new branch] gh/kurtamohler/60/orig -> origin/gh/kurtamohler/60/orig 2025-12-04T09:20:54.4342284Z * [new branch] gh/kurtamohler/61/base -> origin/gh/kurtamohler/61/base 2025-12-04T09:20:54.4342443Z * [new branch] gh/kurtamohler/61/head -> origin/gh/kurtamohler/61/head 2025-12-04T09:20:54.4342595Z * [new branch] gh/kurtamohler/61/orig -> origin/gh/kurtamohler/61/orig 2025-12-04T09:20:54.4346676Z * [new branch] gh/kurtamohler/62/base -> origin/gh/kurtamohler/62/base 2025-12-04T09:20:54.4346862Z * [new branch] gh/kurtamohler/62/head -> origin/gh/kurtamohler/62/head 2025-12-04T09:20:54.4347020Z * [new branch] gh/kurtamohler/62/orig -> origin/gh/kurtamohler/62/orig 2025-12-04T09:20:54.4347166Z * [new branch] gh/kurtamohler/63/base -> origin/gh/kurtamohler/63/base 2025-12-04T09:20:54.4347318Z * [new branch] gh/kurtamohler/63/head -> origin/gh/kurtamohler/63/head 2025-12-04T09:20:54.4352595Z * [new branch] gh/kurtamohler/63/orig -> origin/gh/kurtamohler/63/orig 2025-12-04T09:20:54.4352793Z * [new branch] gh/kurtamohler/64/base -> origin/gh/kurtamohler/64/base 2025-12-04T09:20:54.4352950Z * [new branch] gh/kurtamohler/64/head -> origin/gh/kurtamohler/64/head 2025-12-04T09:20:54.4353251Z * [new branch] gh/kurtamohler/64/orig -> origin/gh/kurtamohler/64/orig 2025-12-04T09:20:54.4353415Z * [new branch] gh/kurtamohler/65/base -> origin/gh/kurtamohler/65/base 2025-12-04T09:20:54.4353550Z * [new branch] gh/kurtamohler/65/head -> origin/gh/kurtamohler/65/head 2025-12-04T09:20:54.4353692Z * [new branch] gh/kurtamohler/65/orig -> origin/gh/kurtamohler/65/orig 2025-12-04T09:20:54.4353940Z * [new branch] gh/kurtamohler/66/base -> origin/gh/kurtamohler/66/base 2025-12-04T09:20:54.4354071Z * [new branch] gh/kurtamohler/66/head -> origin/gh/kurtamohler/66/head 2025-12-04T09:20:54.4354214Z * [new branch] gh/kurtamohler/66/orig -> origin/gh/kurtamohler/66/orig 2025-12-04T09:20:54.4354394Z * [new branch] gh/kurtamohler/67/base -> origin/gh/kurtamohler/67/base 2025-12-04T09:20:54.4354643Z * [new branch] gh/kurtamohler/67/head -> origin/gh/kurtamohler/67/head 2025-12-04T09:20:54.4354821Z * [new branch] gh/kurtamohler/67/orig -> origin/gh/kurtamohler/67/orig 2025-12-04T09:20:54.4355720Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-12-04T09:20:54.4355861Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-12-04T09:20:54.4356274Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-12-04T09:20:54.4358655Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-12-04T09:20:54.4358829Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-12-04T09:20:54.4358981Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-12-04T09:20:54.4359865Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-12-04T09:20:54.4360090Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-12-04T09:20:54.4364479Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-12-04T09:20:54.4364669Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-12-04T09:20:54.4364803Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-12-04T09:20:54.4364938Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-12-04T09:20:54.4365068Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-12-04T09:20:54.4365664Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-12-04T09:20:54.4366094Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-12-04T09:20:54.4366260Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-12-04T09:20:54.4367652Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-12-04T09:20:54.4367992Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-12-04T09:20:54.4371062Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-12-04T09:20:54.4371368Z * [new branch] gh/kwen2501/234/base -> origin/gh/kwen2501/234/base 2025-12-04T09:20:54.4371716Z * [new branch] gh/kwen2501/234/head -> origin/gh/kwen2501/234/head 2025-12-04T09:20:54.4371866Z * [new branch] gh/kwen2501/234/orig -> origin/gh/kwen2501/234/orig 2025-12-04T09:20:54.4372012Z * [new branch] gh/kwen2501/235/base -> origin/gh/kwen2501/235/base 2025-12-04T09:20:54.4376978Z * [new branch] gh/kwen2501/235/head -> origin/gh/kwen2501/235/head 2025-12-04T09:20:54.4382114Z * [new branch] gh/kwen2501/235/orig -> origin/gh/kwen2501/235/orig 2025-12-04T09:20:54.4382293Z * [new branch] gh/kwen2501/236/base -> origin/gh/kwen2501/236/base 2025-12-04T09:20:54.4382766Z * [new branch] gh/kwen2501/236/head -> origin/gh/kwen2501/236/head 2025-12-04T09:20:54.4382945Z * [new branch] gh/kwen2501/236/orig -> origin/gh/kwen2501/236/orig 2025-12-04T09:20:54.4387126Z * [new branch] gh/kwen2501/237/base -> origin/gh/kwen2501/237/base 2025-12-04T09:20:54.4387500Z * [new branch] gh/kwen2501/237/head -> origin/gh/kwen2501/237/head 2025-12-04T09:20:54.4387702Z * [new branch] gh/kwen2501/237/orig -> origin/gh/kwen2501/237/orig 2025-12-04T09:20:54.4388132Z * [new branch] gh/kwen2501/238/base -> origin/gh/kwen2501/238/base 2025-12-04T09:20:54.4388405Z * [new branch] gh/kwen2501/238/head -> origin/gh/kwen2501/238/head 2025-12-04T09:20:54.4388936Z * [new branch] gh/kwen2501/238/orig -> origin/gh/kwen2501/238/orig 2025-12-04T09:20:54.4389331Z * [new branch] gh/kwen2501/240/base -> origin/gh/kwen2501/240/base 2025-12-04T09:20:54.4389497Z * [new branch] gh/kwen2501/240/head -> origin/gh/kwen2501/240/head 2025-12-04T09:20:54.4389641Z * [new branch] gh/kwen2501/240/orig -> origin/gh/kwen2501/240/orig 2025-12-04T09:20:54.4389782Z * [new branch] gh/kwen2501/241/base -> origin/gh/kwen2501/241/base 2025-12-04T09:20:54.4389938Z * [new branch] gh/kwen2501/241/head -> origin/gh/kwen2501/241/head 2025-12-04T09:20:54.4390065Z * [new branch] gh/kwen2501/241/orig -> origin/gh/kwen2501/241/orig 2025-12-04T09:20:54.4390199Z * [new branch] gh/kwen2501/247/base -> origin/gh/kwen2501/247/base 2025-12-04T09:20:54.4396200Z * [new branch] gh/kwen2501/247/head -> origin/gh/kwen2501/247/head 2025-12-04T09:20:54.4396377Z * [new branch] gh/kwen2501/247/orig -> origin/gh/kwen2501/247/orig 2025-12-04T09:20:54.4396532Z * [new branch] gh/kwen2501/252/base -> origin/gh/kwen2501/252/base 2025-12-04T09:20:54.4396662Z * [new branch] gh/kwen2501/252/head -> origin/gh/kwen2501/252/head 2025-12-04T09:20:54.4396797Z * [new branch] gh/kwen2501/252/orig -> origin/gh/kwen2501/252/orig 2025-12-04T09:20:54.4396942Z * [new branch] gh/kwen2501/259/base -> origin/gh/kwen2501/259/base 2025-12-04T09:20:54.4401699Z * [new branch] gh/kwen2501/259/head -> origin/gh/kwen2501/259/head 2025-12-04T09:20:54.4402034Z * [new branch] gh/kwen2501/259/orig -> origin/gh/kwen2501/259/orig 2025-12-04T09:20:54.4402304Z * [new branch] gh/kwen2501/260/base -> origin/gh/kwen2501/260/base 2025-12-04T09:20:54.4402547Z * [new branch] gh/kwen2501/260/head -> origin/gh/kwen2501/260/head 2025-12-04T09:20:54.4402706Z * [new branch] gh/kwen2501/260/orig -> origin/gh/kwen2501/260/orig 2025-12-04T09:20:54.4402908Z * [new branch] gh/kwen2501/268/base -> origin/gh/kwen2501/268/base 2025-12-04T09:20:54.4403115Z * [new branch] gh/kwen2501/268/head -> origin/gh/kwen2501/268/head 2025-12-04T09:20:54.4403773Z * [new branch] gh/kwen2501/268/orig -> origin/gh/kwen2501/268/orig 2025-12-04T09:20:54.4404115Z * [new branch] gh/kwen2501/269/base -> origin/gh/kwen2501/269/base 2025-12-04T09:20:54.4404286Z * [new branch] gh/kwen2501/269/head -> origin/gh/kwen2501/269/head 2025-12-04T09:20:54.4404430Z * [new branch] gh/kwen2501/269/orig -> origin/gh/kwen2501/269/orig 2025-12-04T09:20:54.4406915Z * [new branch] gh/kwen2501/270/base -> origin/gh/kwen2501/270/base 2025-12-04T09:20:54.4407096Z * [new branch] gh/kwen2501/270/head -> origin/gh/kwen2501/270/head 2025-12-04T09:20:54.4407369Z * [new branch] gh/kwen2501/270/orig -> origin/gh/kwen2501/270/orig 2025-12-04T09:20:54.4407537Z * [new branch] gh/kwen2501/271/base -> origin/gh/kwen2501/271/base 2025-12-04T09:20:54.4407681Z * [new branch] gh/kwen2501/271/head -> origin/gh/kwen2501/271/head 2025-12-04T09:20:54.4407812Z * [new branch] gh/kwen2501/271/orig -> origin/gh/kwen2501/271/orig 2025-12-04T09:20:54.4408302Z * [new branch] gh/kwen2501/274/base -> origin/gh/kwen2501/274/base 2025-12-04T09:20:54.4409311Z * [new branch] gh/kwen2501/274/head -> origin/gh/kwen2501/274/head 2025-12-04T09:20:54.4409744Z * [new branch] gh/kwen2501/274/orig -> origin/gh/kwen2501/274/orig 2025-12-04T09:20:54.4411095Z * [new branch] gh/kwen2501/275/base -> origin/gh/kwen2501/275/base 2025-12-04T09:20:54.4411831Z * [new branch] gh/kwen2501/275/head -> origin/gh/kwen2501/275/head 2025-12-04T09:20:54.4417325Z * [new branch] gh/kwen2501/275/orig -> origin/gh/kwen2501/275/orig 2025-12-04T09:20:54.4417685Z * [new branch] gh/kwen2501/276/base -> origin/gh/kwen2501/276/base 2025-12-04T09:20:54.4418013Z * [new branch] gh/kwen2501/276/head -> origin/gh/kwen2501/276/head 2025-12-04T09:20:54.4418161Z * [new branch] gh/kwen2501/276/orig -> origin/gh/kwen2501/276/orig 2025-12-04T09:20:54.4418295Z * [new branch] gh/kwen2501/277/base -> origin/gh/kwen2501/277/base 2025-12-04T09:20:54.4418428Z * [new branch] gh/kwen2501/277/head -> origin/gh/kwen2501/277/head 2025-12-04T09:20:54.4418567Z * [new branch] gh/kwen2501/277/orig -> origin/gh/kwen2501/277/orig 2025-12-04T09:20:54.4421066Z * [new branch] gh/kwen2501/278/base -> origin/gh/kwen2501/278/base 2025-12-04T09:20:54.4421221Z * [new branch] gh/kwen2501/278/head -> origin/gh/kwen2501/278/head 2025-12-04T09:20:54.4421474Z * [new branch] gh/kwen2501/278/orig -> origin/gh/kwen2501/278/orig 2025-12-04T09:20:54.4421623Z * [new branch] gh/kwen2501/279/base -> origin/gh/kwen2501/279/base 2025-12-04T09:20:54.4421850Z * [new branch] gh/kwen2501/279/head -> origin/gh/kwen2501/279/head 2025-12-04T09:20:54.4422035Z * [new branch] gh/kwen2501/279/orig -> origin/gh/kwen2501/279/orig 2025-12-04T09:20:54.4426642Z * [new branch] gh/kwen2501/280/base -> origin/gh/kwen2501/280/base 2025-12-04T09:20:54.4426799Z * [new branch] gh/kwen2501/280/head -> origin/gh/kwen2501/280/head 2025-12-04T09:20:54.4426952Z * [new branch] gh/kwen2501/280/orig -> origin/gh/kwen2501/280/orig 2025-12-04T09:20:54.4427113Z * [new branch] gh/kwen2501/281/base -> origin/gh/kwen2501/281/base 2025-12-04T09:20:54.4427281Z * [new branch] gh/kwen2501/281/head -> origin/gh/kwen2501/281/head 2025-12-04T09:20:54.4427439Z * [new branch] gh/kwen2501/281/orig -> origin/gh/kwen2501/281/orig 2025-12-04T09:20:54.4427602Z * [new branch] gh/kwen2501/282/base -> origin/gh/kwen2501/282/base 2025-12-04T09:20:54.4435313Z * [new branch] gh/kwen2501/282/head -> origin/gh/kwen2501/282/head 2025-12-04T09:20:54.4435667Z * [new branch] gh/kwen2501/282/orig -> origin/gh/kwen2501/282/orig 2025-12-04T09:20:54.4435841Z * [new branch] gh/kwen2501/283/base -> origin/gh/kwen2501/283/base 2025-12-04T09:20:54.4435992Z * [new branch] gh/kwen2501/283/head -> origin/gh/kwen2501/283/head 2025-12-04T09:20:54.4436154Z * [new branch] gh/kwen2501/283/orig -> origin/gh/kwen2501/283/orig 2025-12-04T09:20:54.4436305Z * [new branch] gh/kwen2501/284/base -> origin/gh/kwen2501/284/base 2025-12-04T09:20:54.4436595Z * [new branch] gh/kwen2501/284/head -> origin/gh/kwen2501/284/head 2025-12-04T09:20:54.4474170Z * [new branch] gh/kwen2501/284/orig -> origin/gh/kwen2501/284/orig 2025-12-04T09:20:54.4474432Z * [new branch] gh/kwen2501/285/base -> origin/gh/kwen2501/285/base 2025-12-04T09:20:54.4474575Z * [new branch] gh/kwen2501/285/head -> origin/gh/kwen2501/285/head 2025-12-04T09:20:54.4474710Z * [new branch] gh/kwen2501/285/orig -> origin/gh/kwen2501/285/orig 2025-12-04T09:20:54.4474836Z * [new branch] gh/kwen2501/286/base -> origin/gh/kwen2501/286/base 2025-12-04T09:20:54.4474961Z * [new branch] gh/kwen2501/286/head -> origin/gh/kwen2501/286/head 2025-12-04T09:20:54.4475671Z * [new branch] gh/kwen2501/286/orig -> origin/gh/kwen2501/286/orig 2025-12-04T09:20:54.4475869Z * [new branch] gh/kwen2501/287/base -> origin/gh/kwen2501/287/base 2025-12-04T09:20:54.4476076Z * [new branch] gh/kwen2501/287/head -> origin/gh/kwen2501/287/head 2025-12-04T09:20:54.4476281Z * [new branch] gh/kwen2501/287/orig -> origin/gh/kwen2501/287/orig 2025-12-04T09:20:54.4476797Z * [new branch] gh/kwen2501/288/base -> origin/gh/kwen2501/288/base 2025-12-04T09:20:54.4476970Z * [new branch] gh/kwen2501/288/head -> origin/gh/kwen2501/288/head 2025-12-04T09:20:54.4477113Z * [new branch] gh/kwen2501/288/orig -> origin/gh/kwen2501/288/orig 2025-12-04T09:20:54.4477261Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-12-04T09:20:54.4477425Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-12-04T09:20:54.4477560Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-12-04T09:20:54.4477694Z * [new branch] gh/laithsakka/276/base -> origin/gh/laithsakka/276/base 2025-12-04T09:20:54.4477843Z * [new branch] gh/laithsakka/276/head -> origin/gh/laithsakka/276/head 2025-12-04T09:20:54.4477973Z * [new branch] gh/laithsakka/276/orig -> origin/gh/laithsakka/276/orig 2025-12-04T09:20:54.4478119Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-12-04T09:20:54.4478255Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-12-04T09:20:54.4478384Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-12-04T09:20:54.4478521Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-12-04T09:20:54.4478658Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-12-04T09:20:54.4478789Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-12-04T09:20:54.4478933Z * [new branch] gh/laithsakka/313/base -> origin/gh/laithsakka/313/base 2025-12-04T09:20:54.4479069Z * [new branch] gh/laithsakka/313/head -> origin/gh/laithsakka/313/head 2025-12-04T09:20:54.4479211Z * [new branch] gh/laithsakka/313/orig -> origin/gh/laithsakka/313/orig 2025-12-04T09:20:54.4479343Z * [new branch] gh/laithsakka/316/base -> origin/gh/laithsakka/316/base 2025-12-04T09:20:54.4479475Z * [new branch] gh/laithsakka/316/head -> origin/gh/laithsakka/316/head 2025-12-04T09:20:54.4479614Z * [new branch] gh/laithsakka/316/orig -> origin/gh/laithsakka/316/orig 2025-12-04T09:20:54.4479744Z * [new branch] gh/laithsakka/317/base -> origin/gh/laithsakka/317/base 2025-12-04T09:20:54.4479888Z * [new branch] gh/laithsakka/317/head -> origin/gh/laithsakka/317/head 2025-12-04T09:20:54.4480021Z * [new branch] gh/laithsakka/317/orig -> origin/gh/laithsakka/317/orig 2025-12-04T09:20:54.4480154Z * [new branch] gh/laithsakka/319/base -> origin/gh/laithsakka/319/base 2025-12-04T09:20:54.4480297Z * [new branch] gh/laithsakka/319/head -> origin/gh/laithsakka/319/head 2025-12-04T09:20:54.4480429Z * [new branch] gh/laithsakka/319/orig -> origin/gh/laithsakka/319/orig 2025-12-04T09:20:54.4480566Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-12-04T09:20:54.4480697Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-12-04T09:20:54.4480828Z * [new branch] gh/laithsakka/320/base -> origin/gh/laithsakka/320/base 2025-12-04T09:20:54.4481114Z * [new branch] gh/laithsakka/320/head -> origin/gh/laithsakka/320/head 2025-12-04T09:20:54.4481246Z * [new branch] gh/laithsakka/320/orig -> origin/gh/laithsakka/320/orig 2025-12-04T09:20:54.4481377Z * [new branch] gh/laithsakka/321/base -> origin/gh/laithsakka/321/base 2025-12-04T09:20:54.4481564Z * [new branch] gh/laithsakka/321/head -> origin/gh/laithsakka/321/head 2025-12-04T09:20:54.4481698Z * [new branch] gh/laithsakka/321/orig -> origin/gh/laithsakka/321/orig 2025-12-04T09:20:54.4481851Z * [new branch] gh/laithsakka/322/base -> origin/gh/laithsakka/322/base 2025-12-04T09:20:54.4481985Z * [new branch] gh/laithsakka/322/head -> origin/gh/laithsakka/322/head 2025-12-04T09:20:54.4482124Z * [new branch] gh/laithsakka/322/orig -> origin/gh/laithsakka/322/orig 2025-12-04T09:20:54.4482254Z * [new branch] gh/laithsakka/323/base -> origin/gh/laithsakka/323/base 2025-12-04T09:20:54.4482390Z * [new branch] gh/laithsakka/323/head -> origin/gh/laithsakka/323/head 2025-12-04T09:20:54.4482528Z * [new branch] gh/laithsakka/323/orig -> origin/gh/laithsakka/323/orig 2025-12-04T09:20:54.4482660Z * [new branch] gh/laithsakka/324/base -> origin/gh/laithsakka/324/base 2025-12-04T09:20:54.4482793Z * [new branch] gh/laithsakka/324/head -> origin/gh/laithsakka/324/head 2025-12-04T09:20:54.4482935Z * [new branch] gh/laithsakka/324/orig -> origin/gh/laithsakka/324/orig 2025-12-04T09:20:54.4483066Z * [new branch] gh/laithsakka/325/base -> origin/gh/laithsakka/325/base 2025-12-04T09:20:54.4483206Z * [new branch] gh/laithsakka/325/head -> origin/gh/laithsakka/325/head 2025-12-04T09:20:54.4483451Z * [new branch] gh/laithsakka/325/orig -> origin/gh/laithsakka/325/orig 2025-12-04T09:20:54.4483630Z * [new branch] gh/laithsakka/326/base -> origin/gh/laithsakka/326/base 2025-12-04T09:20:54.4483771Z * [new branch] gh/laithsakka/326/head -> origin/gh/laithsakka/326/head 2025-12-04T09:20:54.4483901Z * [new branch] gh/laithsakka/326/orig -> origin/gh/laithsakka/326/orig 2025-12-04T09:20:54.4484048Z * [new branch] gh/laithsakka/327/base -> origin/gh/laithsakka/327/base 2025-12-04T09:20:54.4484180Z * [new branch] gh/laithsakka/327/head -> origin/gh/laithsakka/327/head 2025-12-04T09:20:54.4484313Z * [new branch] gh/laithsakka/327/orig -> origin/gh/laithsakka/327/orig 2025-12-04T09:20:54.4484453Z * [new branch] gh/laithsakka/328/base -> origin/gh/laithsakka/328/base 2025-12-04T09:20:54.4484585Z * [new branch] gh/laithsakka/328/head -> origin/gh/laithsakka/328/head 2025-12-04T09:20:54.4484727Z * [new branch] gh/laithsakka/328/orig -> origin/gh/laithsakka/328/orig 2025-12-04T09:20:54.4484866Z * [new branch] gh/liangel/4/base -> origin/gh/liangel/4/base 2025-12-04T09:20:54.4484996Z * [new branch] gh/liangel/4/head -> origin/gh/liangel/4/head 2025-12-04T09:20:54.4485127Z * [new branch] gh/liangel/4/orig -> origin/gh/liangel/4/orig 2025-12-04T09:20:54.4486205Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-12-04T09:20:54.4486688Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-12-04T09:20:54.4489260Z * [new branch] gh/lw/4/base -> origin/gh/lw/4/base 2025-12-04T09:20:54.4489413Z * [new branch] gh/lw/4/head -> origin/gh/lw/4/head 2025-12-04T09:20:54.4489541Z * [new branch] gh/lw/4/orig -> origin/gh/lw/4/orig 2025-12-04T09:20:54.4490124Z * [new branch] gh/lw/5/base -> origin/gh/lw/5/base 2025-12-04T09:20:54.4490826Z * [new branch] gh/lw/5/head -> origin/gh/lw/5/head 2025-12-04T09:20:54.4491927Z * [new branch] gh/lw/5/orig -> origin/gh/lw/5/orig 2025-12-04T09:20:54.4493199Z * [new branch] gh/lw/6/base -> origin/gh/lw/6/base 2025-12-04T09:20:54.4493625Z * [new branch] gh/lw/6/head -> origin/gh/lw/6/head 2025-12-04T09:20:54.4494010Z * [new branch] gh/lw/6/orig -> origin/gh/lw/6/orig 2025-12-04T09:20:54.4497836Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-12-04T09:20:54.4498161Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-12-04T09:20:54.4498324Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-12-04T09:20:54.4498455Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-12-04T09:20:54.4498741Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-12-04T09:20:54.4499053Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-12-04T09:20:54.4499882Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-12-04T09:20:54.4502701Z * [new branch] gh/malfet/517/base -> origin/gh/malfet/517/base 2025-12-04T09:20:54.4503015Z * [new branch] gh/malfet/517/head -> origin/gh/malfet/517/head 2025-12-04T09:20:54.4503271Z * [new branch] gh/malfet/528/base -> origin/gh/malfet/528/base 2025-12-04T09:20:54.4503416Z * [new branch] gh/malfet/528/head -> origin/gh/malfet/528/head 2025-12-04T09:20:54.4503560Z * [new branch] gh/malfet/528/orig -> origin/gh/malfet/528/orig 2025-12-04T09:20:54.4505739Z * [new branch] gh/malfet/537/base -> origin/gh/malfet/537/base 2025-12-04T09:20:54.4506087Z * [new branch] gh/malfet/537/head -> origin/gh/malfet/537/head 2025-12-04T09:20:54.4506248Z * [new branch] gh/malfet/537/orig -> origin/gh/malfet/537/orig 2025-12-04T09:20:54.4506574Z * [new branch] gh/malfet/546/base -> origin/gh/malfet/546/base 2025-12-04T09:20:54.4507969Z * [new branch] gh/malfet/546/head -> origin/gh/malfet/546/head 2025-12-04T09:20:54.4508137Z * [new branch] gh/malfet/546/orig -> origin/gh/malfet/546/orig 2025-12-04T09:20:54.4509970Z * [new branch] gh/malfet/565/base -> origin/gh/malfet/565/base 2025-12-04T09:20:54.4510168Z * [new branch] gh/malfet/565/head -> origin/gh/malfet/565/head 2025-12-04T09:20:54.4510436Z * [new branch] gh/malfet/565/orig -> origin/gh/malfet/565/orig 2025-12-04T09:20:54.4512377Z * [new branch] gh/malfet/575/base -> origin/gh/malfet/575/base 2025-12-04T09:20:54.4512732Z * [new branch] gh/malfet/575/head -> origin/gh/malfet/575/head 2025-12-04T09:20:54.4512931Z * [new branch] gh/malfet/575/orig -> origin/gh/malfet/575/orig 2025-12-04T09:20:54.4518366Z * [new branch] gh/malfet/580/base -> origin/gh/malfet/580/base 2025-12-04T09:20:54.4518713Z * [new branch] gh/malfet/580/head -> origin/gh/malfet/580/head 2025-12-04T09:20:54.4518871Z * [new branch] gh/malfet/580/orig -> origin/gh/malfet/580/orig 2025-12-04T09:20:54.4518996Z * [new branch] gh/malfet/581/base -> origin/gh/malfet/581/base 2025-12-04T09:20:54.4519257Z * [new branch] gh/malfet/581/head -> origin/gh/malfet/581/head 2025-12-04T09:20:54.4519467Z * [new branch] gh/malfet/581/orig -> origin/gh/malfet/581/orig 2025-12-04T09:20:54.4523392Z * [new branch] gh/malfet/583/base -> origin/gh/malfet/583/base 2025-12-04T09:20:54.4523881Z * [new branch] gh/malfet/583/head -> origin/gh/malfet/583/head 2025-12-04T09:20:54.4524061Z * [new branch] gh/malfet/583/orig -> origin/gh/malfet/583/orig 2025-12-04T09:20:54.4524234Z * [new branch] gh/malfet/586/base -> origin/gh/malfet/586/base 2025-12-04T09:20:54.4524481Z * [new branch] gh/malfet/586/head -> origin/gh/malfet/586/head 2025-12-04T09:20:54.4524709Z * [new branch] gh/malfet/586/orig -> origin/gh/malfet/586/orig 2025-12-04T09:20:54.4524884Z * [new branch] gh/malfet/587/base -> origin/gh/malfet/587/base 2025-12-04T09:20:54.4525407Z * [new branch] gh/malfet/587/head -> origin/gh/malfet/587/head 2025-12-04T09:20:54.4525539Z * [new branch] gh/malfet/587/orig -> origin/gh/malfet/587/orig 2025-12-04T09:20:54.4525666Z * [new branch] gh/malfet/588/base -> origin/gh/malfet/588/base 2025-12-04T09:20:54.4525809Z * [new branch] gh/malfet/588/head -> origin/gh/malfet/588/head 2025-12-04T09:20:54.4525934Z * [new branch] gh/malfet/588/orig -> origin/gh/malfet/588/orig 2025-12-04T09:20:54.4526087Z * [new branch] gh/malfet/589/base -> origin/gh/malfet/589/base 2025-12-04T09:20:54.4526419Z * [new branch] gh/malfet/589/head -> origin/gh/malfet/589/head 2025-12-04T09:20:54.4526703Z * [new branch] gh/malfet/589/orig -> origin/gh/malfet/589/orig 2025-12-04T09:20:54.4527804Z * [new branch] gh/malfet/590/base -> origin/gh/malfet/590/base 2025-12-04T09:20:54.4528130Z * [new branch] gh/malfet/590/head -> origin/gh/malfet/590/head 2025-12-04T09:20:54.4530170Z * [new branch] gh/malfet/590/orig -> origin/gh/malfet/590/orig 2025-12-04T09:20:54.4530338Z * [new branch] gh/malfet/591/base -> origin/gh/malfet/591/base 2025-12-04T09:20:54.4530634Z * [new branch] gh/malfet/591/head -> origin/gh/malfet/591/head 2025-12-04T09:20:54.4531801Z * [new branch] gh/malfet/591/orig -> origin/gh/malfet/591/orig 2025-12-04T09:20:54.4532683Z * [new branch] gh/malfet/592/base -> origin/gh/malfet/592/base 2025-12-04T09:20:54.4533251Z * [new branch] gh/malfet/592/head -> origin/gh/malfet/592/head 2025-12-04T09:20:54.4534215Z * [new branch] gh/malfet/592/orig -> origin/gh/malfet/592/orig 2025-12-04T09:20:54.4535089Z * [new branch] gh/malfet/593/base -> origin/gh/malfet/593/base 2025-12-04T09:20:54.4536140Z * [new branch] gh/malfet/593/head -> origin/gh/malfet/593/head 2025-12-04T09:20:54.4536618Z * [new branch] gh/malfet/593/orig -> origin/gh/malfet/593/orig 2025-12-04T09:20:54.4537841Z * [new branch] gh/malfet/594/base -> origin/gh/malfet/594/base 2025-12-04T09:20:54.4538164Z * [new branch] gh/malfet/594/head -> origin/gh/malfet/594/head 2025-12-04T09:20:54.4539174Z * [new branch] gh/malfet/594/orig -> origin/gh/malfet/594/orig 2025-12-04T09:20:54.4540158Z * [new branch] gh/malfet/595/base -> origin/gh/malfet/595/base 2025-12-04T09:20:54.4540497Z * [new branch] gh/malfet/595/head -> origin/gh/malfet/595/head 2025-12-04T09:20:54.4541857Z * [new branch] gh/malfet/595/orig -> origin/gh/malfet/595/orig 2025-12-04T09:20:54.4542005Z * [new branch] gh/malfet/596/base -> origin/gh/malfet/596/base 2025-12-04T09:20:54.4543997Z * [new branch] gh/malfet/596/head -> origin/gh/malfet/596/head 2025-12-04T09:20:54.4544169Z * [new branch] gh/malfet/596/orig -> origin/gh/malfet/596/orig 2025-12-04T09:20:54.4544348Z * [new branch] gh/malfet/597/base -> origin/gh/malfet/597/base 2025-12-04T09:20:54.4550604Z * [new branch] gh/malfet/597/head -> origin/gh/malfet/597/head 2025-12-04T09:20:54.4550778Z * [new branch] gh/malfet/597/orig -> origin/gh/malfet/597/orig 2025-12-04T09:20:54.4551038Z * [new branch] gh/malfet/598/base -> origin/gh/malfet/598/base 2025-12-04T09:20:54.4551249Z * [new branch] gh/malfet/598/head -> origin/gh/malfet/598/head 2025-12-04T09:20:54.4551384Z * [new branch] gh/malfet/598/orig -> origin/gh/malfet/598/orig 2025-12-04T09:20:54.4551519Z * [new branch] gh/malfet/599/base -> origin/gh/malfet/599/base 2025-12-04T09:20:54.4551644Z * [new branch] gh/malfet/599/head -> origin/gh/malfet/599/head 2025-12-04T09:20:54.4551766Z * [new branch] gh/malfet/599/orig -> origin/gh/malfet/599/orig 2025-12-04T09:20:54.4551905Z * [new branch] gh/malfet/600/base -> origin/gh/malfet/600/base 2025-12-04T09:20:54.4552034Z * [new branch] gh/malfet/600/head -> origin/gh/malfet/600/head 2025-12-04T09:20:54.4552259Z * [new branch] gh/malfet/600/orig -> origin/gh/malfet/600/orig 2025-12-04T09:20:54.4557912Z * [new branch] gh/malfet/601/base -> origin/gh/malfet/601/base 2025-12-04T09:20:54.4558220Z * [new branch] gh/malfet/601/head -> origin/gh/malfet/601/head 2025-12-04T09:20:54.4558393Z * [new branch] gh/malfet/601/orig -> origin/gh/malfet/601/orig 2025-12-04T09:20:54.4558541Z * [new branch] gh/malfet/602/base -> origin/gh/malfet/602/base 2025-12-04T09:20:54.4558679Z * [new branch] gh/malfet/602/head -> origin/gh/malfet/602/head 2025-12-04T09:20:54.4558820Z * [new branch] gh/malfet/602/orig -> origin/gh/malfet/602/orig 2025-12-04T09:20:54.4561166Z * [new branch] gh/malfet/603/base -> origin/gh/malfet/603/base 2025-12-04T09:20:54.4561337Z * [new branch] gh/malfet/603/head -> origin/gh/malfet/603/head 2025-12-04T09:20:54.4561795Z * [new branch] gh/malfet/603/orig -> origin/gh/malfet/603/orig 2025-12-04T09:20:54.4561979Z * [new branch] gh/malfet/604/base -> origin/gh/malfet/604/base 2025-12-04T09:20:54.4562120Z * [new branch] gh/malfet/604/head -> origin/gh/malfet/604/head 2025-12-04T09:20:54.4562245Z * [new branch] gh/malfet/604/orig -> origin/gh/malfet/604/orig 2025-12-04T09:20:54.4562381Z * [new branch] gh/malfet/605/base -> origin/gh/malfet/605/base 2025-12-04T09:20:54.4565025Z * [new branch] gh/malfet/605/head -> origin/gh/malfet/605/head 2025-12-04T09:20:54.4565217Z * [new branch] gh/malfet/605/orig -> origin/gh/malfet/605/orig 2025-12-04T09:20:54.4565375Z * [new branch] gh/malfet/606/base -> origin/gh/malfet/606/base 2025-12-04T09:20:54.4565637Z * [new branch] gh/malfet/606/head -> origin/gh/malfet/606/head 2025-12-04T09:20:54.4565847Z * [new branch] gh/malfet/606/orig -> origin/gh/malfet/606/orig 2025-12-04T09:20:54.4565981Z * [new branch] gh/malfet/607/base -> origin/gh/malfet/607/base 2025-12-04T09:20:54.4566133Z * [new branch] gh/malfet/607/head -> origin/gh/malfet/607/head 2025-12-04T09:20:54.4567862Z * [new branch] gh/malfet/607/orig -> origin/gh/malfet/607/orig 2025-12-04T09:20:54.4568040Z * [new branch] gh/malfet/608/base -> origin/gh/malfet/608/base 2025-12-04T09:20:54.4568501Z * [new branch] gh/malfet/608/head -> origin/gh/malfet/608/head 2025-12-04T09:20:54.4570949Z * [new branch] gh/malfet/608/orig -> origin/gh/malfet/608/orig 2025-12-04T09:20:54.4571300Z * [new branch] gh/malfet/609/base -> origin/gh/malfet/609/base 2025-12-04T09:20:54.4571615Z * [new branch] gh/malfet/609/head -> origin/gh/malfet/609/head 2025-12-04T09:20:54.4571785Z * [new branch] gh/malfet/609/orig -> origin/gh/malfet/609/orig 2025-12-04T09:20:54.4576885Z * [new branch] gh/malfet/610/base -> origin/gh/malfet/610/base 2025-12-04T09:20:54.4577078Z * [new branch] gh/malfet/610/head -> origin/gh/malfet/610/head 2025-12-04T09:20:54.4577232Z * [new branch] gh/malfet/610/orig -> origin/gh/malfet/610/orig 2025-12-04T09:20:54.4577374Z * [new branch] gh/malfet/611/base -> origin/gh/malfet/611/base 2025-12-04T09:20:54.4577824Z * [new branch] gh/malfet/611/head -> origin/gh/malfet/611/head 2025-12-04T09:20:54.4577978Z * [new branch] gh/malfet/611/orig -> origin/gh/malfet/611/orig 2025-12-04T09:20:54.4578129Z * [new branch] gh/malfet/612/base -> origin/gh/malfet/612/base 2025-12-04T09:20:54.4578292Z * [new branch] gh/malfet/612/head -> origin/gh/malfet/612/head 2025-12-04T09:20:54.4584191Z * [new branch] gh/malfet/612/orig -> origin/gh/malfet/612/orig 2025-12-04T09:20:54.4584539Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-12-04T09:20:54.4584780Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-12-04T09:20:54.4584975Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-12-04T09:20:54.4585287Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-12-04T09:20:54.4589603Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-12-04T09:20:54.4589941Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-12-04T09:20:54.4590195Z * [new branch] gh/masnesral/1/base -> origin/gh/masnesral/1/base 2025-12-04T09:20:54.4590354Z * [new branch] gh/masnesral/1/head -> origin/gh/masnesral/1/head 2025-12-04T09:20:54.4590605Z * [new branch] gh/masnesral/1/orig -> origin/gh/masnesral/1/orig 2025-12-04T09:20:54.4591287Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-12-04T09:20:54.4591656Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-12-04T09:20:54.4592030Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-12-04T09:20:54.4592293Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-12-04T09:20:54.4592440Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-12-04T09:20:54.4592585Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-12-04T09:20:54.4597877Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-12-04T09:20:54.4598206Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-12-04T09:20:54.4598376Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-12-04T09:20:54.4598612Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-12-04T09:20:54.4598783Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-12-04T09:20:54.4598914Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-12-04T09:20:54.4599050Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-12-04T09:20:54.4599316Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-12-04T09:20:54.4599960Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-12-04T09:20:54.4600138Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-12-04T09:20:54.4600299Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-12-04T09:20:54.4600520Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-12-04T09:20:54.4600811Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-12-04T09:20:54.4601144Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-12-04T09:20:54.4606832Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-12-04T09:20:54.4607204Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-12-04T09:20:54.4607396Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-12-04T09:20:54.4607689Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-12-04T09:20:54.4608074Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-12-04T09:20:54.4608484Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-12-04T09:20:54.4608679Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-12-04T09:20:54.4608863Z * [new branch] gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base 2025-12-04T09:20:54.4609032Z * [new branch] gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head 2025-12-04T09:20:54.4609207Z * [new branch] gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig 2025-12-04T09:20:54.4609379Z * [new branch] gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base 2025-12-04T09:20:54.4609555Z * [new branch] gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head 2025-12-04T09:20:54.4610538Z * [new branch] gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig 2025-12-04T09:20:54.4611688Z * [new branch] gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base 2025-12-04T09:20:54.4612205Z * [new branch] gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head 2025-12-04T09:20:54.4612683Z * [new branch] gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig 2025-12-04T09:20:54.4616943Z * [new branch] gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base 2025-12-04T09:20:54.4617312Z * [new branch] gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head 2025-12-04T09:20:54.4617515Z * [new branch] gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig 2025-12-04T09:20:54.4617699Z * [new branch] gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base 2025-12-04T09:20:54.4617906Z * [new branch] gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head 2025-12-04T09:20:54.4618206Z * [new branch] gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig 2025-12-04T09:20:54.4618498Z * [new branch] gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base 2025-12-04T09:20:54.4618966Z * [new branch] gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head 2025-12-04T09:20:54.4621205Z * [new branch] gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig 2025-12-04T09:20:54.4621457Z * [new branch] gh/mikaylagawarecki/351/base -> origin/gh/mikaylagawarecki/351/base 2025-12-04T09:20:54.4621848Z * [new branch] gh/mikaylagawarecki/351/head -> origin/gh/mikaylagawarecki/351/head 2025-12-04T09:20:54.4622511Z * [new branch] gh/mikaylagawarecki/351/orig -> origin/gh/mikaylagawarecki/351/orig 2025-12-04T09:20:54.4624168Z * [new branch] gh/mikaylagawarecki/352/base -> origin/gh/mikaylagawarecki/352/base 2025-12-04T09:20:54.4624584Z * [new branch] gh/mikaylagawarecki/352/head -> origin/gh/mikaylagawarecki/352/head 2025-12-04T09:20:54.4625180Z * [new branch] gh/mikaylagawarecki/352/orig -> origin/gh/mikaylagawarecki/352/orig 2025-12-04T09:20:54.4628262Z * [new branch] gh/mikaylagawarecki/353/base -> origin/gh/mikaylagawarecki/353/base 2025-12-04T09:20:54.4633003Z * [new branch] gh/mikaylagawarecki/353/head -> origin/gh/mikaylagawarecki/353/head 2025-12-04T09:20:54.4636997Z * [new branch] gh/mikaylagawarecki/353/orig -> origin/gh/mikaylagawarecki/353/orig 2025-12-04T09:20:54.4641396Z * [new branch] gh/mikaylagawarecki/354/base -> origin/gh/mikaylagawarecki/354/base 2025-12-04T09:20:54.4643385Z * [new branch] gh/mikaylagawarecki/354/head -> origin/gh/mikaylagawarecki/354/head 2025-12-04T09:20:54.4643579Z * [new branch] gh/mikaylagawarecki/354/orig -> origin/gh/mikaylagawarecki/354/orig 2025-12-04T09:20:54.4643762Z * [new branch] gh/mikaylagawarecki/356/base -> origin/gh/mikaylagawarecki/356/base 2025-12-04T09:20:54.4643947Z * [new branch] gh/mikaylagawarecki/356/head -> origin/gh/mikaylagawarecki/356/head 2025-12-04T09:20:54.4644110Z * [new branch] gh/mikaylagawarecki/356/orig -> origin/gh/mikaylagawarecki/356/orig 2025-12-04T09:20:54.4644275Z * [new branch] gh/mikaylagawarecki/357/base -> origin/gh/mikaylagawarecki/357/base 2025-12-04T09:20:54.4644447Z * [new branch] gh/mikaylagawarecki/357/head -> origin/gh/mikaylagawarecki/357/head 2025-12-04T09:20:54.4644612Z * [new branch] gh/mikaylagawarecki/357/orig -> origin/gh/mikaylagawarecki/357/orig 2025-12-04T09:20:54.4644778Z * [new branch] gh/mikaylagawarecki/359/base -> origin/gh/mikaylagawarecki/359/base 2025-12-04T09:20:54.4644935Z * [new branch] gh/mikaylagawarecki/359/head -> origin/gh/mikaylagawarecki/359/head 2025-12-04T09:20:54.4645092Z * [new branch] gh/mikaylagawarecki/359/orig -> origin/gh/mikaylagawarecki/359/orig 2025-12-04T09:20:54.4645267Z * [new branch] gh/mikaylagawarecki/360/base -> origin/gh/mikaylagawarecki/360/base 2025-12-04T09:20:54.4645424Z * [new branch] gh/mikaylagawarecki/360/head -> origin/gh/mikaylagawarecki/360/head 2025-12-04T09:20:54.4645588Z * [new branch] gh/mikaylagawarecki/360/orig -> origin/gh/mikaylagawarecki/360/orig 2025-12-04T09:20:54.4645746Z * [new branch] gh/mikaylagawarecki/361/base -> origin/gh/mikaylagawarecki/361/base 2025-12-04T09:20:54.4645903Z * [new branch] gh/mikaylagawarecki/361/head -> origin/gh/mikaylagawarecki/361/head 2025-12-04T09:20:54.4646084Z * [new branch] gh/mikaylagawarecki/361/orig -> origin/gh/mikaylagawarecki/361/orig 2025-12-04T09:20:54.4646267Z * [new branch] gh/mikaylagawarecki/362/base -> origin/gh/mikaylagawarecki/362/base 2025-12-04T09:20:54.4646440Z * [new branch] gh/mikaylagawarecki/362/head -> origin/gh/mikaylagawarecki/362/head 2025-12-04T09:20:54.4646615Z * [new branch] gh/mikaylagawarecki/362/orig -> origin/gh/mikaylagawarecki/362/orig 2025-12-04T09:20:54.4646794Z * [new branch] gh/mikaylagawarecki/363/base -> origin/gh/mikaylagawarecki/363/base 2025-12-04T09:20:54.4646979Z * [new branch] gh/mikaylagawarecki/363/head -> origin/gh/mikaylagawarecki/363/head 2025-12-04T09:20:54.4647163Z * [new branch] gh/mikaylagawarecki/363/orig -> origin/gh/mikaylagawarecki/363/orig 2025-12-04T09:20:54.4648411Z * [new branch] gh/mikaylagawarecki/364/base -> origin/gh/mikaylagawarecki/364/base 2025-12-04T09:20:54.4648823Z * [new branch] gh/mikaylagawarecki/364/head -> origin/gh/mikaylagawarecki/364/head 2025-12-04T09:20:54.4651695Z * [new branch] gh/mikaylagawarecki/364/orig -> origin/gh/mikaylagawarecki/364/orig 2025-12-04T09:20:54.4651922Z * [new branch] gh/mikaylagawarecki/365/base -> origin/gh/mikaylagawarecki/365/base 2025-12-04T09:20:54.4652322Z * [new branch] gh/mikaylagawarecki/365/head -> origin/gh/mikaylagawarecki/365/head 2025-12-04T09:20:54.4656675Z * [new branch] gh/mikaylagawarecki/365/orig -> origin/gh/mikaylagawarecki/365/orig 2025-12-04T09:20:54.4660981Z * [new branch] gh/mikaylagawarecki/366/base -> origin/gh/mikaylagawarecki/366/base 2025-12-04T09:20:54.4665195Z * [new branch] gh/mikaylagawarecki/366/head -> origin/gh/mikaylagawarecki/366/head 2025-12-04T09:20:54.4669428Z * [new branch] gh/mikaylagawarecki/366/orig -> origin/gh/mikaylagawarecki/366/orig 2025-12-04T09:20:54.4669652Z * [new branch] gh/mikaylagawarecki/367/base -> origin/gh/mikaylagawarecki/367/base 2025-12-04T09:20:54.4669862Z * [new branch] gh/mikaylagawarecki/367/head -> origin/gh/mikaylagawarecki/367/head 2025-12-04T09:20:54.4670041Z * [new branch] gh/mikaylagawarecki/367/orig -> origin/gh/mikaylagawarecki/367/orig 2025-12-04T09:20:54.4670215Z * [new branch] gh/mikaylagawarecki/368/base -> origin/gh/mikaylagawarecki/368/base 2025-12-04T09:20:54.4670385Z * [new branch] gh/mikaylagawarecki/368/head -> origin/gh/mikaylagawarecki/368/head 2025-12-04T09:20:54.4670567Z * [new branch] gh/mikaylagawarecki/368/orig -> origin/gh/mikaylagawarecki/368/orig 2025-12-04T09:20:54.4670739Z * [new branch] gh/mikaylagawarecki/369/base -> origin/gh/mikaylagawarecki/369/base 2025-12-04T09:20:54.4670913Z * [new branch] gh/mikaylagawarecki/369/head -> origin/gh/mikaylagawarecki/369/head 2025-12-04T09:20:54.4671079Z * [new branch] gh/mikaylagawarecki/369/orig -> origin/gh/mikaylagawarecki/369/orig 2025-12-04T09:20:54.4671250Z * [new branch] gh/mikaylagawarecki/370/base -> origin/gh/mikaylagawarecki/370/base 2025-12-04T09:20:54.4671422Z * [new branch] gh/mikaylagawarecki/370/head -> origin/gh/mikaylagawarecki/370/head 2025-12-04T09:20:54.4671594Z * [new branch] gh/mikaylagawarecki/370/orig -> origin/gh/mikaylagawarecki/370/orig 2025-12-04T09:20:54.4671769Z * [new branch] gh/mikaylagawarecki/371/base -> origin/gh/mikaylagawarecki/371/base 2025-12-04T09:20:54.4671935Z * [new branch] gh/mikaylagawarecki/371/head -> origin/gh/mikaylagawarecki/371/head 2025-12-04T09:20:54.4672104Z * [new branch] gh/mikaylagawarecki/371/orig -> origin/gh/mikaylagawarecki/371/orig 2025-12-04T09:20:54.4672475Z * [new branch] gh/mikaylagawarecki/372/base -> origin/gh/mikaylagawarecki/372/base 2025-12-04T09:20:54.4672656Z * [new branch] gh/mikaylagawarecki/372/head -> origin/gh/mikaylagawarecki/372/head 2025-12-04T09:20:54.4672834Z * [new branch] gh/mikaylagawarecki/372/orig -> origin/gh/mikaylagawarecki/372/orig 2025-12-04T09:20:54.4673000Z * [new branch] gh/mikaylagawarecki/373/base -> origin/gh/mikaylagawarecki/373/base 2025-12-04T09:20:54.4673167Z * [new branch] gh/mikaylagawarecki/373/head -> origin/gh/mikaylagawarecki/373/head 2025-12-04T09:20:54.4677222Z * [new branch] gh/mikaylagawarecki/373/orig -> origin/gh/mikaylagawarecki/373/orig 2025-12-04T09:20:54.4677446Z * [new branch] gh/mikaylagawarecki/374/base -> origin/gh/mikaylagawarecki/374/base 2025-12-04T09:20:54.4677634Z * [new branch] gh/mikaylagawarecki/374/head -> origin/gh/mikaylagawarecki/374/head 2025-12-04T09:20:54.4677807Z * [new branch] gh/mikaylagawarecki/374/orig -> origin/gh/mikaylagawarecki/374/orig 2025-12-04T09:20:54.4677977Z * [new branch] gh/mikaylagawarecki/375/base -> origin/gh/mikaylagawarecki/375/base 2025-12-04T09:20:54.4678393Z * [new branch] gh/mikaylagawarecki/375/head -> origin/gh/mikaylagawarecki/375/head 2025-12-04T09:20:54.4678565Z * [new branch] gh/mikaylagawarecki/375/orig -> origin/gh/mikaylagawarecki/375/orig 2025-12-04T09:20:54.4683845Z * [new branch] gh/mikaylagawarecki/376/base -> origin/gh/mikaylagawarecki/376/base 2025-12-04T09:20:54.4684491Z * [new branch] gh/mikaylagawarecki/376/head -> origin/gh/mikaylagawarecki/376/head 2025-12-04T09:20:54.4684691Z * [new branch] gh/mikaylagawarecki/376/orig -> origin/gh/mikaylagawarecki/376/orig 2025-12-04T09:20:54.4684864Z * [new branch] gh/mikaylagawarecki/377/base -> origin/gh/mikaylagawarecki/377/base 2025-12-04T09:20:54.4685063Z * [new branch] gh/mikaylagawarecki/377/head -> origin/gh/mikaylagawarecki/377/head 2025-12-04T09:20:54.4685240Z * [new branch] gh/mikaylagawarecki/377/orig -> origin/gh/mikaylagawarecki/377/orig 2025-12-04T09:20:54.4685419Z * [new branch] gh/mikaylagawarecki/378/base -> origin/gh/mikaylagawarecki/378/base 2025-12-04T09:20:54.4685602Z * [new branch] gh/mikaylagawarecki/378/head -> origin/gh/mikaylagawarecki/378/head 2025-12-04T09:20:54.4685770Z * [new branch] gh/mikaylagawarecki/378/orig -> origin/gh/mikaylagawarecki/378/orig 2025-12-04T09:20:54.4685946Z * [new branch] gh/mikaylagawarecki/379/base -> origin/gh/mikaylagawarecki/379/base 2025-12-04T09:20:54.4686121Z * [new branch] gh/mikaylagawarecki/379/head -> origin/gh/mikaylagawarecki/379/head 2025-12-04T09:20:54.4686315Z * [new branch] gh/mikaylagawarecki/379/orig -> origin/gh/mikaylagawarecki/379/orig 2025-12-04T09:20:54.4690141Z * [new branch] gh/mikaylagawarecki/380/base -> origin/gh/mikaylagawarecki/380/base 2025-12-04T09:20:54.4690356Z * [new branch] gh/mikaylagawarecki/380/head -> origin/gh/mikaylagawarecki/380/head 2025-12-04T09:20:54.4690567Z * [new branch] gh/mikaylagawarecki/380/orig -> origin/gh/mikaylagawarecki/380/orig 2025-12-04T09:20:54.4690740Z * [new branch] gh/mikaylagawarecki/381/base -> origin/gh/mikaylagawarecki/381/base 2025-12-04T09:20:54.4690916Z * [new branch] gh/mikaylagawarecki/381/head -> origin/gh/mikaylagawarecki/381/head 2025-12-04T09:20:54.4691320Z * [new branch] gh/mikaylagawarecki/381/orig -> origin/gh/mikaylagawarecki/381/orig 2025-12-04T09:20:54.4691866Z * [new branch] gh/mikaylagawarecki/382/base -> origin/gh/mikaylagawarecki/382/base 2025-12-04T09:20:54.4692838Z * [new branch] gh/mikaylagawarecki/382/head -> origin/gh/mikaylagawarecki/382/head 2025-12-04T09:20:54.4693266Z * [new branch] gh/mikaylagawarecki/382/orig -> origin/gh/mikaylagawarecki/382/orig 2025-12-04T09:20:54.4696999Z * [new branch] gh/mikaylagawarecki/383/base -> origin/gh/mikaylagawarecki/383/base 2025-12-04T09:20:54.4697235Z * [new branch] gh/mikaylagawarecki/383/head -> origin/gh/mikaylagawarecki/383/head 2025-12-04T09:20:54.4697416Z * [new branch] gh/mikaylagawarecki/383/orig -> origin/gh/mikaylagawarecki/383/orig 2025-12-04T09:20:54.4697585Z * [new branch] gh/mikaylagawarecki/384/base -> origin/gh/mikaylagawarecki/384/base 2025-12-04T09:20:54.4697787Z * [new branch] gh/mikaylagawarecki/384/head -> origin/gh/mikaylagawarecki/384/head 2025-12-04T09:20:54.4698231Z * [new branch] gh/mikaylagawarecki/384/orig -> origin/gh/mikaylagawarecki/384/orig 2025-12-04T09:20:54.4699447Z * [new branch] gh/mikaylagawarecki/385/base -> origin/gh/mikaylagawarecki/385/base 2025-12-04T09:20:54.4700031Z * [new branch] gh/mikaylagawarecki/385/head -> origin/gh/mikaylagawarecki/385/head 2025-12-04T09:20:54.4700757Z * [new branch] gh/mikaylagawarecki/385/orig -> origin/gh/mikaylagawarecki/385/orig 2025-12-04T09:20:54.4702120Z * [new branch] gh/mikaylagawarecki/386/base -> origin/gh/mikaylagawarecki/386/base 2025-12-04T09:20:54.4702354Z * [new branch] gh/mikaylagawarecki/386/head -> origin/gh/mikaylagawarecki/386/head 2025-12-04T09:20:54.4703452Z * [new branch] gh/mikaylagawarecki/386/orig -> origin/gh/mikaylagawarecki/386/orig 2025-12-04T09:20:54.4704466Z * [new branch] gh/mikaylagawarecki/387/base -> origin/gh/mikaylagawarecki/387/base 2025-12-04T09:20:54.4704731Z * [new branch] gh/mikaylagawarecki/387/head -> origin/gh/mikaylagawarecki/387/head 2025-12-04T09:20:54.4705807Z * [new branch] gh/mikaylagawarecki/387/orig -> origin/gh/mikaylagawarecki/387/orig 2025-12-04T09:20:54.4706420Z * [new branch] gh/mikaylagawarecki/388/base -> origin/gh/mikaylagawarecki/388/base 2025-12-04T09:20:54.4707419Z * [new branch] gh/mikaylagawarecki/388/head -> origin/gh/mikaylagawarecki/388/head 2025-12-04T09:20:54.4707726Z * [new branch] gh/mikaylagawarecki/388/orig -> origin/gh/mikaylagawarecki/388/orig 2025-12-04T09:20:54.4709211Z * [new branch] gh/mikaylagawarecki/389/base -> origin/gh/mikaylagawarecki/389/base 2025-12-04T09:20:54.4709723Z * [new branch] gh/mikaylagawarecki/389/head -> origin/gh/mikaylagawarecki/389/head 2025-12-04T09:20:54.4711004Z * [new branch] gh/mikaylagawarecki/389/orig -> origin/gh/mikaylagawarecki/389/orig 2025-12-04T09:20:54.4711472Z * [new branch] gh/mikaylagawarecki/390/base -> origin/gh/mikaylagawarecki/390/base 2025-12-04T09:20:54.4712370Z * [new branch] gh/mikaylagawarecki/390/head -> origin/gh/mikaylagawarecki/390/head 2025-12-04T09:20:54.4712723Z * [new branch] gh/mikaylagawarecki/390/orig -> origin/gh/mikaylagawarecki/390/orig 2025-12-04T09:20:54.4714316Z * [new branch] gh/mikaylagawarecki/391/base -> origin/gh/mikaylagawarecki/391/base 2025-12-04T09:20:54.4714643Z * [new branch] gh/mikaylagawarecki/391/head -> origin/gh/mikaylagawarecki/391/head 2025-12-04T09:20:54.4715772Z * [new branch] gh/mikaylagawarecki/391/orig -> origin/gh/mikaylagawarecki/391/orig 2025-12-04T09:20:54.4716619Z * [new branch] gh/mikaylagawarecki/392/base -> origin/gh/mikaylagawarecki/392/base 2025-12-04T09:20:54.4717125Z * [new branch] gh/mikaylagawarecki/392/head -> origin/gh/mikaylagawarecki/392/head 2025-12-04T09:20:54.4718123Z * [new branch] gh/mikaylagawarecki/392/orig -> origin/gh/mikaylagawarecki/392/orig 2025-12-04T09:20:54.4719405Z * [new branch] gh/mlazos/41/base -> origin/gh/mlazos/41/base 2025-12-04T09:20:54.4719800Z * [new branch] gh/mlazos/41/head -> origin/gh/mlazos/41/head 2025-12-04T09:20:54.4720867Z * [new branch] gh/mlazos/41/orig -> origin/gh/mlazos/41/orig 2025-12-04T09:20:54.4721735Z * [new branch] gh/mlazos/42/base -> origin/gh/mlazos/42/base 2025-12-04T09:20:54.4722870Z * [new branch] gh/mlazos/42/head -> origin/gh/mlazos/42/head 2025-12-04T09:20:54.4723340Z * [new branch] gh/mlazos/42/orig -> origin/gh/mlazos/42/orig 2025-12-04T09:20:54.4723501Z * [new branch] gh/mlazos/43/base -> origin/gh/mlazos/43/base 2025-12-04T09:20:54.4724554Z * [new branch] gh/mlazos/43/head -> origin/gh/mlazos/43/head 2025-12-04T09:20:54.4724730Z * [new branch] gh/mlazos/43/orig -> origin/gh/mlazos/43/orig 2025-12-04T09:20:54.4726009Z * [new branch] gh/mlazos/44/base -> origin/gh/mlazos/44/base 2025-12-04T09:20:54.4729914Z * [new branch] gh/mlazos/44/head -> origin/gh/mlazos/44/head 2025-12-04T09:20:54.4730152Z * [new branch] gh/mlazos/44/orig -> origin/gh/mlazos/44/orig 2025-12-04T09:20:54.4730301Z * [new branch] gh/mlazos/47/base -> origin/gh/mlazos/47/base 2025-12-04T09:20:54.4730519Z * [new branch] gh/mlazos/47/head -> origin/gh/mlazos/47/head 2025-12-04T09:20:54.4730664Z * [new branch] gh/mlazos/47/orig -> origin/gh/mlazos/47/orig 2025-12-04T09:20:54.4730803Z * [new branch] gh/mlazos/48/base -> origin/gh/mlazos/48/base 2025-12-04T09:20:54.4731207Z * [new branch] gh/mlazos/48/head -> origin/gh/mlazos/48/head 2025-12-04T09:20:54.4736095Z * [new branch] gh/mlazos/48/orig -> origin/gh/mlazos/48/orig 2025-12-04T09:20:54.4736264Z * [new branch] gh/mlazos/49/base -> origin/gh/mlazos/49/base 2025-12-04T09:20:54.4736410Z * [new branch] gh/mlazos/49/head -> origin/gh/mlazos/49/head 2025-12-04T09:20:54.4736549Z * [new branch] gh/mlazos/49/orig -> origin/gh/mlazos/49/orig 2025-12-04T09:20:54.4736690Z * [new branch] gh/mlazos/50/base -> origin/gh/mlazos/50/base 2025-12-04T09:20:54.4742565Z * [new branch] gh/mlazos/50/head -> origin/gh/mlazos/50/head 2025-12-04T09:20:54.4747450Z * [new branch] gh/mlazos/50/orig -> origin/gh/mlazos/50/orig 2025-12-04T09:20:54.4752471Z * [new branch] gh/mlazos/51/base -> origin/gh/mlazos/51/base 2025-12-04T09:20:54.4756920Z * [new branch] gh/mlazos/51/head -> origin/gh/mlazos/51/head 2025-12-04T09:20:54.4761228Z * [new branch] gh/mlazos/51/orig -> origin/gh/mlazos/51/orig 2025-12-04T09:20:54.4761423Z * [new branch] gh/mlazos/52/base -> origin/gh/mlazos/52/base 2025-12-04T09:20:54.4761569Z * [new branch] gh/mlazos/52/head -> origin/gh/mlazos/52/head 2025-12-04T09:20:54.4761714Z * [new branch] gh/mlazos/52/orig -> origin/gh/mlazos/52/orig 2025-12-04T09:20:54.4761856Z * [new branch] gh/mlazos/53/base -> origin/gh/mlazos/53/base 2025-12-04T09:20:54.4762014Z * [new branch] gh/mlazos/53/head -> origin/gh/mlazos/53/head 2025-12-04T09:20:54.4762146Z * [new branch] gh/mlazos/53/orig -> origin/gh/mlazos/53/orig 2025-12-04T09:20:54.4762287Z * [new branch] gh/mlazos/54/base -> origin/gh/mlazos/54/base 2025-12-04T09:20:54.4762425Z * [new branch] gh/mlazos/54/head -> origin/gh/mlazos/54/head 2025-12-04T09:20:54.4762563Z * [new branch] gh/mlazos/54/orig -> origin/gh/mlazos/54/orig 2025-12-04T09:20:54.4762689Z * [new branch] gh/mlazos/55/base -> origin/gh/mlazos/55/base 2025-12-04T09:20:54.4762815Z * [new branch] gh/mlazos/55/head -> origin/gh/mlazos/55/head 2025-12-04T09:20:54.4762953Z * [new branch] gh/mlazos/55/orig -> origin/gh/mlazos/55/orig 2025-12-04T09:20:54.4763083Z * [new branch] gh/mlazos/56/base -> origin/gh/mlazos/56/base 2025-12-04T09:20:54.4763220Z * [new branch] gh/mlazos/56/head -> origin/gh/mlazos/56/head 2025-12-04T09:20:54.4763359Z * [new branch] gh/mlazos/56/orig -> origin/gh/mlazos/56/orig 2025-12-04T09:20:54.4763506Z * [new branch] gh/mlazos/57/base -> origin/gh/mlazos/57/base 2025-12-04T09:20:54.4763648Z * [new branch] gh/mlazos/57/head -> origin/gh/mlazos/57/head 2025-12-04T09:20:54.4763778Z * [new branch] gh/mlazos/57/orig -> origin/gh/mlazos/57/orig 2025-12-04T09:20:54.4763907Z * [new branch] gh/mlazos/58/base -> origin/gh/mlazos/58/base 2025-12-04T09:20:54.4764043Z * [new branch] gh/mlazos/58/head -> origin/gh/mlazos/58/head 2025-12-04T09:20:54.4764174Z * [new branch] gh/mlazos/58/orig -> origin/gh/mlazos/58/orig 2025-12-04T09:20:54.4764310Z * [new branch] gh/mlazos/59/base -> origin/gh/mlazos/59/base 2025-12-04T09:20:54.4764609Z * [new branch] gh/mlazos/59/head -> origin/gh/mlazos/59/head 2025-12-04T09:20:54.4764743Z * [new branch] gh/mlazos/59/orig -> origin/gh/mlazos/59/orig 2025-12-04T09:20:54.4764883Z * [new branch] gh/mlazos/60/base -> origin/gh/mlazos/60/base 2025-12-04T09:20:54.4765068Z * [new branch] gh/mlazos/60/head -> origin/gh/mlazos/60/head 2025-12-04T09:20:54.4765215Z * [new branch] gh/mlazos/60/orig -> origin/gh/mlazos/60/orig 2025-12-04T09:20:54.4765348Z * [new branch] gh/mlazos/61/base -> origin/gh/mlazos/61/base 2025-12-04T09:20:54.4765483Z * [new branch] gh/mlazos/61/head -> origin/gh/mlazos/61/head 2025-12-04T09:20:54.4765629Z * [new branch] gh/mlazos/61/orig -> origin/gh/mlazos/61/orig 2025-12-04T09:20:54.4765762Z * [new branch] gh/mlazos/62/base -> origin/gh/mlazos/62/base 2025-12-04T09:20:54.4765903Z * [new branch] gh/mlazos/62/head -> origin/gh/mlazos/62/head 2025-12-04T09:20:54.4766032Z * [new branch] gh/mlazos/62/orig -> origin/gh/mlazos/62/orig 2025-12-04T09:20:54.4766161Z * [new branch] gh/mlazos/63/base -> origin/gh/mlazos/63/base 2025-12-04T09:20:54.4766300Z * [new branch] gh/mlazos/63/head -> origin/gh/mlazos/63/head 2025-12-04T09:20:54.4766435Z * [new branch] gh/mlazos/63/orig -> origin/gh/mlazos/63/orig 2025-12-04T09:20:54.4766994Z * [new branch] gh/mlazos/64/base -> origin/gh/mlazos/64/base 2025-12-04T09:20:54.4767791Z * [new branch] gh/mlazos/64/head -> origin/gh/mlazos/64/head 2025-12-04T09:20:54.4768685Z * [new branch] gh/mlazos/64/orig -> origin/gh/mlazos/64/orig 2025-12-04T09:20:54.4769705Z * [new branch] gh/mlazos/65/base -> origin/gh/mlazos/65/base 2025-12-04T09:20:54.4770198Z * [new branch] gh/mlazos/65/head -> origin/gh/mlazos/65/head 2025-12-04T09:20:54.4770873Z * [new branch] gh/mlazos/65/orig -> origin/gh/mlazos/65/orig 2025-12-04T09:20:54.4773796Z * [new branch] gh/mlazos/66/base -> origin/gh/mlazos/66/base 2025-12-04T09:20:54.4774156Z * [new branch] gh/mlazos/66/head -> origin/gh/mlazos/66/head 2025-12-04T09:20:54.4780119Z * [new branch] gh/mlazos/66/orig -> origin/gh/mlazos/66/orig 2025-12-04T09:20:54.4780447Z * [new branch] gh/mlazos/67/base -> origin/gh/mlazos/67/base 2025-12-04T09:20:54.4780780Z * [new branch] gh/mlazos/67/head -> origin/gh/mlazos/67/head 2025-12-04T09:20:54.4780929Z * [new branch] gh/mlazos/67/orig -> origin/gh/mlazos/67/orig 2025-12-04T09:20:54.4786217Z * [new branch] gh/mlazos/68/base -> origin/gh/mlazos/68/base 2025-12-04T09:20:54.4788468Z * [new branch] gh/mlazos/68/head -> origin/gh/mlazos/68/head 2025-12-04T09:20:54.4788773Z * [new branch] gh/mlazos/68/orig -> origin/gh/mlazos/68/orig 2025-12-04T09:20:54.4794114Z * [new branch] gh/mlazos/69/base -> origin/gh/mlazos/69/base 2025-12-04T09:20:54.4796490Z * [new branch] gh/mlazos/69/head -> origin/gh/mlazos/69/head 2025-12-04T09:20:54.4801443Z * [new branch] gh/mlazos/69/orig -> origin/gh/mlazos/69/orig 2025-12-04T09:20:54.4803463Z * [new branch] gh/mlazos/70/base -> origin/gh/mlazos/70/base 2025-12-04T09:20:54.4803732Z * [new branch] gh/mlazos/70/head -> origin/gh/mlazos/70/head 2025-12-04T09:20:54.4807413Z * [new branch] gh/mlazos/70/orig -> origin/gh/mlazos/70/orig 2025-12-04T09:20:54.4807587Z * [new branch] gh/mlazos/71/base -> origin/gh/mlazos/71/base 2025-12-04T09:20:54.4807957Z * [new branch] gh/mlazos/71/head -> origin/gh/mlazos/71/head 2025-12-04T09:20:54.4808095Z * [new branch] gh/mlazos/71/orig -> origin/gh/mlazos/71/orig 2025-12-04T09:20:54.4808231Z * [new branch] gh/mlazos/72/base -> origin/gh/mlazos/72/base 2025-12-04T09:20:54.4808447Z * [new branch] gh/mlazos/72/head -> origin/gh/mlazos/72/head 2025-12-04T09:20:54.4808607Z * [new branch] gh/mlazos/72/orig -> origin/gh/mlazos/72/orig 2025-12-04T09:20:54.4808734Z * [new branch] gh/mlazos/73/base -> origin/gh/mlazos/73/base 2025-12-04T09:20:54.4808870Z * [new branch] gh/mlazos/73/head -> origin/gh/mlazos/73/head 2025-12-04T09:20:54.4808999Z * [new branch] gh/mlazos/73/orig -> origin/gh/mlazos/73/orig 2025-12-04T09:20:54.4809138Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-12-04T09:20:54.4809292Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-12-04T09:20:54.4809442Z * [new branch] gh/muchulee8/73/base -> origin/gh/muchulee8/73/base 2025-12-04T09:20:54.4809592Z * [new branch] gh/muchulee8/73/head -> origin/gh/muchulee8/73/head 2025-12-04T09:20:54.4809736Z * [new branch] gh/muchulee8/73/orig -> origin/gh/muchulee8/73/orig 2025-12-04T09:20:54.4809907Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-12-04T09:20:54.4810076Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-12-04T09:20:54.4810236Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-12-04T09:20:54.4810386Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-12-04T09:20:54.4810547Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-12-04T09:20:54.4810703Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-12-04T09:20:54.4810863Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-12-04T09:20:54.4811021Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-12-04T09:20:54.4811181Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-12-04T09:20:54.4811357Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-12-04T09:20:54.4811755Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-12-04T09:20:54.4811930Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-12-04T09:20:54.4812093Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-12-04T09:20:54.4812272Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-12-04T09:20:54.4812764Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-12-04T09:20:54.4814534Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-12-04T09:20:54.4814914Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-12-04T09:20:54.4815095Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-12-04T09:20:54.4817168Z * [new branch] gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base 2025-12-04T09:20:54.4817545Z * [new branch] gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head 2025-12-04T09:20:54.4817724Z * [new branch] gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig 2025-12-04T09:20:54.4819890Z * [new branch] gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base 2025-12-04T09:20:54.4820239Z * [new branch] gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head 2025-12-04T09:20:54.4820410Z * [new branch] gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig 2025-12-04T09:20:54.4820885Z * [new branch] gh/naveenthangudu/9/base -> origin/gh/naveenthangudu/9/base 2025-12-04T09:20:54.4825800Z * [new branch] gh/naveenthangudu/9/head -> origin/gh/naveenthangudu/9/head 2025-12-04T09:20:54.4826224Z * [new branch] gh/naveenthangudu/9/orig -> origin/gh/naveenthangudu/9/orig 2025-12-04T09:20:54.4826420Z * [new branch] gh/nikitaved/1/base -> origin/gh/nikitaved/1/base 2025-12-04T09:20:54.4826670Z * [new branch] gh/nikitaved/1/head -> origin/gh/nikitaved/1/head 2025-12-04T09:20:54.4826823Z * [new branch] gh/nikitaved/1/orig -> origin/gh/nikitaved/1/orig 2025-12-04T09:20:54.4826995Z * [new branch] gh/nikitaved/10/base -> origin/gh/nikitaved/10/base 2025-12-04T09:20:54.4827285Z * [new branch] gh/nikitaved/10/head -> origin/gh/nikitaved/10/head 2025-12-04T09:20:54.4834942Z * [new branch] gh/nikitaved/10/orig -> origin/gh/nikitaved/10/orig 2025-12-04T09:20:54.4837131Z * [new branch] gh/nikitaved/11/base -> origin/gh/nikitaved/11/base 2025-12-04T09:20:54.4840566Z * [new branch] gh/nikitaved/11/head -> origin/gh/nikitaved/11/head 2025-12-04T09:20:54.4840883Z * [new branch] gh/nikitaved/11/orig -> origin/gh/nikitaved/11/orig 2025-12-04T09:20:54.4845777Z * [new branch] gh/nikitaved/12/base -> origin/gh/nikitaved/12/base 2025-12-04T09:20:54.4846153Z * [new branch] gh/nikitaved/12/head -> origin/gh/nikitaved/12/head 2025-12-04T09:20:54.4846327Z * [new branch] gh/nikitaved/12/orig -> origin/gh/nikitaved/12/orig 2025-12-04T09:20:54.4846516Z * [new branch] gh/nikitaved/13/base -> origin/gh/nikitaved/13/base 2025-12-04T09:20:54.4846679Z * [new branch] gh/nikitaved/13/head -> origin/gh/nikitaved/13/head 2025-12-04T09:20:54.4846840Z * [new branch] gh/nikitaved/13/orig -> origin/gh/nikitaved/13/orig 2025-12-04T09:20:54.4847020Z * [new branch] gh/nikitaved/14/base -> origin/gh/nikitaved/14/base 2025-12-04T09:20:54.4847686Z * [new branch] gh/nikitaved/14/head -> origin/gh/nikitaved/14/head 2025-12-04T09:20:54.4847872Z * [new branch] gh/nikitaved/14/orig -> origin/gh/nikitaved/14/orig 2025-12-04T09:20:54.4848200Z * [new branch] gh/nikitaved/15/base -> origin/gh/nikitaved/15/base 2025-12-04T09:20:54.4848370Z * [new branch] gh/nikitaved/15/head -> origin/gh/nikitaved/15/head 2025-12-04T09:20:54.4848526Z * [new branch] gh/nikitaved/15/orig -> origin/gh/nikitaved/15/orig 2025-12-04T09:20:54.4848678Z * [new branch] gh/nikitaved/16/base -> origin/gh/nikitaved/16/base 2025-12-04T09:20:54.4848824Z * [new branch] gh/nikitaved/16/head -> origin/gh/nikitaved/16/head 2025-12-04T09:20:54.4848967Z * [new branch] gh/nikitaved/16/orig -> origin/gh/nikitaved/16/orig 2025-12-04T09:20:54.4849128Z * [new branch] gh/nikitaved/2/base -> origin/gh/nikitaved/2/base 2025-12-04T09:20:54.4849280Z * [new branch] gh/nikitaved/2/head -> origin/gh/nikitaved/2/head 2025-12-04T09:20:54.4849429Z * [new branch] gh/nikitaved/2/orig -> origin/gh/nikitaved/2/orig 2025-12-04T09:20:54.4849577Z * [new branch] gh/nikitaved/4/base -> origin/gh/nikitaved/4/base 2025-12-04T09:20:54.4849716Z * [new branch] gh/nikitaved/4/head -> origin/gh/nikitaved/4/head 2025-12-04T09:20:54.4849855Z * [new branch] gh/nikitaved/4/orig -> origin/gh/nikitaved/4/orig 2025-12-04T09:20:54.4850143Z * [new branch] gh/nikitaved/5/base -> origin/gh/nikitaved/5/base 2025-12-04T09:20:54.4850285Z * [new branch] gh/nikitaved/5/head -> origin/gh/nikitaved/5/head 2025-12-04T09:20:54.4850486Z * [new branch] gh/nikitaved/5/orig -> origin/gh/nikitaved/5/orig 2025-12-04T09:20:54.4850638Z * [new branch] gh/nikitaved/6/base -> origin/gh/nikitaved/6/base 2025-12-04T09:20:54.4850782Z * [new branch] gh/nikitaved/6/head -> origin/gh/nikitaved/6/head 2025-12-04T09:20:54.4850942Z * [new branch] gh/nikitaved/6/orig -> origin/gh/nikitaved/6/orig 2025-12-04T09:20:54.4851091Z * [new branch] gh/nikitaved/8/base -> origin/gh/nikitaved/8/base 2025-12-04T09:20:54.4851243Z * [new branch] gh/nikitaved/8/head -> origin/gh/nikitaved/8/head 2025-12-04T09:20:54.4852200Z * [new branch] gh/nikitaved/8/orig -> origin/gh/nikitaved/8/orig 2025-12-04T09:20:54.4855239Z * [new branch] gh/nikitaved/9/base -> origin/gh/nikitaved/9/base 2025-12-04T09:20:54.4855568Z * [new branch] gh/nikitaved/9/head -> origin/gh/nikitaved/9/head 2025-12-04T09:20:54.4855755Z * [new branch] gh/nikitaved/9/orig -> origin/gh/nikitaved/9/orig 2025-12-04T09:20:54.4855899Z * [new branch] gh/oulgen/10/base -> origin/gh/oulgen/10/base 2025-12-04T09:20:54.4856059Z * [new branch] gh/oulgen/10/head -> origin/gh/oulgen/10/head 2025-12-04T09:20:54.4857067Z * [new branch] gh/oulgen/10/orig -> origin/gh/oulgen/10/orig 2025-12-04T09:20:54.4857988Z * [new branch] gh/oulgen/11/base -> origin/gh/oulgen/11/base 2025-12-04T09:20:54.4858417Z * [new branch] gh/oulgen/11/head -> origin/gh/oulgen/11/head 2025-12-04T09:20:54.4859403Z * [new branch] gh/oulgen/11/orig -> origin/gh/oulgen/11/orig 2025-12-04T09:20:54.4860346Z * [new branch] gh/oulgen/12/base -> origin/gh/oulgen/12/base 2025-12-04T09:20:54.4861612Z * [new branch] gh/oulgen/12/head -> origin/gh/oulgen/12/head 2025-12-04T09:20:54.4862239Z * [new branch] gh/oulgen/12/orig -> origin/gh/oulgen/12/orig 2025-12-04T09:20:54.4864604Z * [new branch] gh/oulgen/13/base -> origin/gh/oulgen/13/base 2025-12-04T09:20:54.4864781Z * [new branch] gh/oulgen/13/head -> origin/gh/oulgen/13/head 2025-12-04T09:20:54.4864930Z * [new branch] gh/oulgen/13/orig -> origin/gh/oulgen/13/orig 2025-12-04T09:20:54.4865080Z * [new branch] gh/oulgen/14/base -> origin/gh/oulgen/14/base 2025-12-04T09:20:54.4865260Z * [new branch] gh/oulgen/14/head -> origin/gh/oulgen/14/head 2025-12-04T09:20:54.4866111Z * [new branch] gh/oulgen/14/orig -> origin/gh/oulgen/14/orig 2025-12-04T09:20:54.4870615Z * [new branch] gh/oulgen/15/base -> origin/gh/oulgen/15/base 2025-12-04T09:20:54.4870793Z * [new branch] gh/oulgen/15/head -> origin/gh/oulgen/15/head 2025-12-04T09:20:54.4870956Z * [new branch] gh/oulgen/15/orig -> origin/gh/oulgen/15/orig 2025-12-04T09:20:54.4871094Z * [new branch] gh/oulgen/16/base -> origin/gh/oulgen/16/base 2025-12-04T09:20:54.4871234Z * [new branch] gh/oulgen/16/head -> origin/gh/oulgen/16/head 2025-12-04T09:20:54.4871819Z * [new branch] gh/oulgen/16/orig -> origin/gh/oulgen/16/orig 2025-12-04T09:20:54.4871971Z * [new branch] gh/oulgen/17/base -> origin/gh/oulgen/17/base 2025-12-04T09:20:54.4875613Z * [new branch] gh/oulgen/17/head -> origin/gh/oulgen/17/head 2025-12-04T09:20:54.4875986Z * [new branch] gh/oulgen/17/orig -> origin/gh/oulgen/17/orig 2025-12-04T09:20:54.4876125Z * [new branch] gh/oulgen/18/base -> origin/gh/oulgen/18/base 2025-12-04T09:20:54.4876256Z * [new branch] gh/oulgen/18/head -> origin/gh/oulgen/18/head 2025-12-04T09:20:54.4876465Z * [new branch] gh/oulgen/18/orig -> origin/gh/oulgen/18/orig 2025-12-04T09:20:54.4881713Z * [new branch] gh/oulgen/19/base -> origin/gh/oulgen/19/base 2025-12-04T09:20:54.4881893Z * [new branch] gh/oulgen/19/head -> origin/gh/oulgen/19/head 2025-12-04T09:20:54.4882056Z * [new branch] gh/oulgen/19/orig -> origin/gh/oulgen/19/orig 2025-12-04T09:20:54.4882193Z * [new branch] gh/oulgen/20/base -> origin/gh/oulgen/20/base 2025-12-04T09:20:54.4882335Z * [new branch] gh/oulgen/20/head -> origin/gh/oulgen/20/head 2025-12-04T09:20:54.4884180Z * [new branch] gh/oulgen/20/orig -> origin/gh/oulgen/20/orig 2025-12-04T09:20:54.4884485Z * [new branch] gh/oulgen/21/base -> origin/gh/oulgen/21/base 2025-12-04T09:20:54.4884631Z * [new branch] gh/oulgen/21/head -> origin/gh/oulgen/21/head 2025-12-04T09:20:54.4884777Z * [new branch] gh/oulgen/21/orig -> origin/gh/oulgen/21/orig 2025-12-04T09:20:54.4884907Z * [new branch] gh/oulgen/22/base -> origin/gh/oulgen/22/base 2025-12-04T09:20:54.4885042Z * [new branch] gh/oulgen/22/head -> origin/gh/oulgen/22/head 2025-12-04T09:20:54.4885326Z * [new branch] gh/oulgen/22/orig -> origin/gh/oulgen/22/orig 2025-12-04T09:20:54.4886114Z * [new branch] gh/oulgen/23/base -> origin/gh/oulgen/23/base 2025-12-04T09:20:54.4886520Z * [new branch] gh/oulgen/23/head -> origin/gh/oulgen/23/head 2025-12-04T09:20:54.4887426Z * [new branch] gh/oulgen/23/orig -> origin/gh/oulgen/23/orig 2025-12-04T09:20:54.4888377Z * [new branch] gh/oulgen/24/base -> origin/gh/oulgen/24/base 2025-12-04T09:20:54.4888887Z * [new branch] gh/oulgen/24/head -> origin/gh/oulgen/24/head 2025-12-04T09:20:54.4890254Z * [new branch] gh/oulgen/24/orig -> origin/gh/oulgen/24/orig 2025-12-04T09:20:54.4891174Z * [new branch] gh/oulgen/25/base -> origin/gh/oulgen/25/base 2025-12-04T09:20:54.4891778Z * [new branch] gh/oulgen/25/head -> origin/gh/oulgen/25/head 2025-12-04T09:20:54.4892779Z * [new branch] gh/oulgen/25/orig -> origin/gh/oulgen/25/orig 2025-12-04T09:20:54.4894213Z * [new branch] gh/oulgen/26/base -> origin/gh/oulgen/26/base 2025-12-04T09:20:54.4894507Z * [new branch] gh/oulgen/26/head -> origin/gh/oulgen/26/head 2025-12-04T09:20:54.4895663Z * [new branch] gh/oulgen/26/orig -> origin/gh/oulgen/26/orig 2025-12-04T09:20:54.4896548Z * [new branch] gh/oulgen/4/base -> origin/gh/oulgen/4/base 2025-12-04T09:20:54.4896727Z * [new branch] gh/oulgen/4/head -> origin/gh/oulgen/4/head 2025-12-04T09:20:54.4898028Z * [new branch] gh/oulgen/4/orig -> origin/gh/oulgen/4/orig 2025-12-04T09:20:54.4899180Z * [new branch] gh/oulgen/7/base -> origin/gh/oulgen/7/base 2025-12-04T09:20:54.4899848Z * [new branch] gh/oulgen/7/head -> origin/gh/oulgen/7/head 2025-12-04T09:20:54.4903835Z * [new branch] gh/oulgen/7/orig -> origin/gh/oulgen/7/orig 2025-12-04T09:20:54.4903969Z * [new branch] gh/oulgen/8/base -> origin/gh/oulgen/8/base 2025-12-04T09:20:54.4904108Z * [new branch] gh/oulgen/8/head -> origin/gh/oulgen/8/head 2025-12-04T09:20:54.4904287Z * [new branch] gh/oulgen/8/orig -> origin/gh/oulgen/8/orig 2025-12-04T09:20:54.4904423Z * [new branch] gh/oulgen/9/base -> origin/gh/oulgen/9/base 2025-12-04T09:20:54.4907434Z * [new branch] gh/oulgen/9/head -> origin/gh/oulgen/9/head 2025-12-04T09:20:54.4907794Z * [new branch] gh/oulgen/9/orig -> origin/gh/oulgen/9/orig 2025-12-04T09:20:54.4908651Z * [new branch] gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization 2025-12-04T09:20:54.4908836Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-12-04T09:20:54.4913339Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-12-04T09:20:54.4913516Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-12-04T09:20:54.4913656Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-12-04T09:20:54.4913812Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-12-04T09:20:54.4913937Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-12-04T09:20:54.4914073Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-12-04T09:20:54.4917640Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-12-04T09:20:54.4917792Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-12-04T09:20:54.4917936Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-12-04T09:20:54.4918216Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-12-04T09:20:54.4918409Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-12-04T09:20:54.4918557Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-12-04T09:20:54.4918718Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-12-04T09:20:54.4918864Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-12-04T09:20:54.4919996Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-12-04T09:20:54.4920478Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-12-04T09:20:54.4924332Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-12-04T09:20:54.4924509Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-12-04T09:20:54.4925099Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-12-04T09:20:54.4925268Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-12-04T09:20:54.4925406Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-12-04T09:20:54.4925588Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-12-04T09:20:54.4925768Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-12-04T09:20:54.4927464Z * [new branch] gh/pearu/118/base -> origin/gh/pearu/118/base 2025-12-04T09:20:54.4927668Z * [new branch] gh/pearu/118/head -> origin/gh/pearu/118/head 2025-12-04T09:20:54.4928310Z * [new branch] gh/pearu/118/orig -> origin/gh/pearu/118/orig 2025-12-04T09:20:54.4929505Z * [new branch] gh/pearu/119/base -> origin/gh/pearu/119/base 2025-12-04T09:20:54.4929696Z * [new branch] gh/pearu/119/head -> origin/gh/pearu/119/head 2025-12-04T09:20:54.4930887Z * [new branch] gh/pearu/119/orig -> origin/gh/pearu/119/orig 2025-12-04T09:20:54.4931897Z * [new branch] gh/pearu/139/base -> origin/gh/pearu/139/base 2025-12-04T09:20:54.4932352Z * [new branch] gh/pearu/139/head -> origin/gh/pearu/139/head 2025-12-04T09:20:54.4933681Z * [new branch] gh/pearu/139/orig -> origin/gh/pearu/139/orig 2025-12-04T09:20:54.4934188Z * [new branch] gh/pearu/140/base -> origin/gh/pearu/140/base 2025-12-04T09:20:54.4935060Z * [new branch] gh/pearu/140/head -> origin/gh/pearu/140/head 2025-12-04T09:20:54.4935447Z * [new branch] gh/pearu/140/orig -> origin/gh/pearu/140/orig 2025-12-04T09:20:54.4939914Z * [new branch] gh/pearu/142/base -> origin/gh/pearu/142/base 2025-12-04T09:20:54.4940095Z * [new branch] gh/pearu/142/head -> origin/gh/pearu/142/head 2025-12-04T09:20:54.4940245Z * [new branch] gh/pearu/142/orig -> origin/gh/pearu/142/orig 2025-12-04T09:20:54.4940383Z * [new branch] gh/pearu/143/base -> origin/gh/pearu/143/base 2025-12-04T09:20:54.4940543Z * [new branch] gh/pearu/143/head -> origin/gh/pearu/143/head 2025-12-04T09:20:54.4940680Z * [new branch] gh/pearu/143/orig -> origin/gh/pearu/143/orig 2025-12-04T09:20:54.4941026Z * [new branch] gh/pearu/147/base -> origin/gh/pearu/147/base 2025-12-04T09:20:54.4941993Z * [new branch] gh/pearu/147/head -> origin/gh/pearu/147/head 2025-12-04T09:20:54.4942636Z * [new branch] gh/pearu/147/orig -> origin/gh/pearu/147/orig 2025-12-04T09:20:54.4946099Z * [new branch] gh/pearu/149/base -> origin/gh/pearu/149/base 2025-12-04T09:20:54.4946427Z * [new branch] gh/pearu/149/head -> origin/gh/pearu/149/head 2025-12-04T09:20:54.4946647Z * [new branch] gh/pearu/149/orig -> origin/gh/pearu/149/orig 2025-12-04T09:20:54.4946868Z * [new branch] gh/pearu/150/base -> origin/gh/pearu/150/base 2025-12-04T09:20:54.4947203Z * [new branch] gh/pearu/150/head -> origin/gh/pearu/150/head 2025-12-04T09:20:54.4947887Z * [new branch] gh/pearu/150/orig -> origin/gh/pearu/150/orig 2025-12-04T09:20:54.4948898Z * [new branch] gh/pearu/151/base -> origin/gh/pearu/151/base 2025-12-04T09:20:54.4949222Z * [new branch] gh/pearu/151/head -> origin/gh/pearu/151/head 2025-12-04T09:20:54.4950159Z * [new branch] gh/pearu/151/orig -> origin/gh/pearu/151/orig 2025-12-04T09:20:54.4953797Z * [new branch] gh/pearu/152/base -> origin/gh/pearu/152/base 2025-12-04T09:20:54.4953975Z * [new branch] gh/pearu/152/head -> origin/gh/pearu/152/head 2025-12-04T09:20:54.4954105Z * [new branch] gh/pearu/152/orig -> origin/gh/pearu/152/orig 2025-12-04T09:20:54.4954233Z * [new branch] gh/pearu/153/base -> origin/gh/pearu/153/base 2025-12-04T09:20:54.4954387Z * [new branch] gh/pearu/153/head -> origin/gh/pearu/153/head 2025-12-04T09:20:54.4954736Z * [new branch] gh/pearu/153/orig -> origin/gh/pearu/153/orig 2025-12-04T09:20:54.4958149Z * [new branch] gh/pearu/154/base -> origin/gh/pearu/154/base 2025-12-04T09:20:54.4958341Z * [new branch] gh/pearu/154/head -> origin/gh/pearu/154/head 2025-12-04T09:20:54.4958473Z * [new branch] gh/pearu/154/orig -> origin/gh/pearu/154/orig 2025-12-04T09:20:54.4958613Z * [new branch] gh/pearu/155/base -> origin/gh/pearu/155/base 2025-12-04T09:20:54.4958754Z * [new branch] gh/pearu/155/head -> origin/gh/pearu/155/head 2025-12-04T09:20:54.4959467Z * [new branch] gh/pearu/155/orig -> origin/gh/pearu/155/orig 2025-12-04T09:20:54.4960910Z * [new branch] gh/pearu/156/base -> origin/gh/pearu/156/base 2025-12-04T09:20:54.4961247Z * [new branch] gh/pearu/156/head -> origin/gh/pearu/156/head 2025-12-04T09:20:54.4965414Z * [new branch] gh/pearu/156/orig -> origin/gh/pearu/156/orig 2025-12-04T09:20:54.4965599Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-12-04T09:20:54.4966130Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-12-04T09:20:54.4966275Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-12-04T09:20:54.4966427Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-12-04T09:20:54.4966925Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-12-04T09:20:54.4967709Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-12-04T09:20:54.4969147Z * [new branch] gh/pianpwk/21/base -> origin/gh/pianpwk/21/base 2025-12-04T09:20:54.4969563Z * [new branch] gh/pianpwk/21/head -> origin/gh/pianpwk/21/head 2025-12-04T09:20:54.4971163Z * [new branch] gh/pianpwk/28/base -> origin/gh/pianpwk/28/base 2025-12-04T09:20:54.4971741Z * [new branch] gh/pianpwk/28/head -> origin/gh/pianpwk/28/head 2025-12-04T09:20:54.4977152Z * [new branch] gh/pianpwk/28/orig -> origin/gh/pianpwk/28/orig 2025-12-04T09:20:54.4977497Z * [new branch] gh/pianpwk/29/base -> origin/gh/pianpwk/29/base 2025-12-04T09:20:54.4977660Z * [new branch] gh/pianpwk/29/head -> origin/gh/pianpwk/29/head 2025-12-04T09:20:54.4977809Z * [new branch] gh/pianpwk/29/orig -> origin/gh/pianpwk/29/orig 2025-12-04T09:20:54.4982487Z * [new branch] gh/pianpwk/30/base -> origin/gh/pianpwk/30/base 2025-12-04T09:20:54.4982848Z * [new branch] gh/pianpwk/30/head -> origin/gh/pianpwk/30/head 2025-12-04T09:20:54.4983004Z * [new branch] gh/pianpwk/30/orig -> origin/gh/pianpwk/30/orig 2025-12-04T09:20:54.4983157Z * [new branch] gh/pianpwk/31/base -> origin/gh/pianpwk/31/base 2025-12-04T09:20:54.4983302Z * [new branch] gh/pianpwk/31/head -> origin/gh/pianpwk/31/head 2025-12-04T09:20:54.4983618Z * [new branch] gh/pianpwk/31/orig -> origin/gh/pianpwk/31/orig 2025-12-04T09:20:54.4983792Z * [new branch] gh/pianpwk/32/base -> origin/gh/pianpwk/32/base 2025-12-04T09:20:54.4983937Z * [new branch] gh/pianpwk/32/head -> origin/gh/pianpwk/32/head 2025-12-04T09:20:54.4984084Z * [new branch] gh/pianpwk/32/orig -> origin/gh/pianpwk/32/orig 2025-12-04T09:20:54.4984227Z * [new branch] gh/pianpwk/33/base -> origin/gh/pianpwk/33/base 2025-12-04T09:20:54.4984504Z * [new branch] gh/pianpwk/33/head -> origin/gh/pianpwk/33/head 2025-12-04T09:20:54.4991738Z * [new branch] gh/pianpwk/33/orig -> origin/gh/pianpwk/33/orig 2025-12-04T09:20:54.4996754Z * [new branch] gh/pianpwk/34/base -> origin/gh/pianpwk/34/base 2025-12-04T09:20:54.4999772Z * [new branch] gh/pianpwk/34/head -> origin/gh/pianpwk/34/head 2025-12-04T09:20:54.5005427Z * [new branch] gh/pianpwk/34/orig -> origin/gh/pianpwk/34/orig 2025-12-04T09:20:54.5007578Z * [new branch] gh/pianpwk/35/base -> origin/gh/pianpwk/35/base 2025-12-04T09:20:54.5007962Z * [new branch] gh/pianpwk/35/head -> origin/gh/pianpwk/35/head 2025-12-04T09:20:54.5008138Z * [new branch] gh/pianpwk/35/orig -> origin/gh/pianpwk/35/orig 2025-12-04T09:20:54.5008290Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-12-04T09:20:54.5008423Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-12-04T09:20:54.5008793Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-12-04T09:20:54.5008942Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-12-04T09:20:54.5009085Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-12-04T09:20:54.5009305Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-12-04T09:20:54.5009450Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-12-04T09:20:54.5009587Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-12-04T09:20:54.5009860Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-12-04T09:20:54.5010007Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-12-04T09:20:54.5010142Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-12-04T09:20:54.5010288Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-12-04T09:20:54.5010430Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-12-04T09:20:54.5010574Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-12-04T09:20:54.5010720Z * [new branch] gh/rec/167/base -> origin/gh/rec/167/base 2025-12-04T09:20:54.5010850Z * [new branch] gh/rec/167/head -> origin/gh/rec/167/head 2025-12-04T09:20:54.5010997Z * [new branch] gh/rec/167/orig -> origin/gh/rec/167/orig 2025-12-04T09:20:54.5011136Z * [new branch] gh/rec/168/base -> origin/gh/rec/168/base 2025-12-04T09:20:54.5011280Z * [new branch] gh/rec/168/head -> origin/gh/rec/168/head 2025-12-04T09:20:54.5011419Z * [new branch] gh/rec/168/orig -> origin/gh/rec/168/orig 2025-12-04T09:20:54.5011620Z * [new branch] gh/rec/169/base -> origin/gh/rec/169/base 2025-12-04T09:20:54.5011765Z * [new branch] gh/rec/169/head -> origin/gh/rec/169/head 2025-12-04T09:20:54.5011903Z * [new branch] gh/rec/169/orig -> origin/gh/rec/169/orig 2025-12-04T09:20:54.5012048Z * [new branch] gh/rec/170/base -> origin/gh/rec/170/base 2025-12-04T09:20:54.5012173Z * [new branch] gh/rec/170/head -> origin/gh/rec/170/head 2025-12-04T09:20:54.5012311Z * [new branch] gh/rec/170/orig -> origin/gh/rec/170/orig 2025-12-04T09:20:54.5012456Z * [new branch] gh/rec/171/base -> origin/gh/rec/171/base 2025-12-04T09:20:54.5012581Z * [new branch] gh/rec/171/head -> origin/gh/rec/171/head 2025-12-04T09:20:54.5012711Z * [new branch] gh/rec/171/orig -> origin/gh/rec/171/orig 2025-12-04T09:20:54.5012846Z * [new branch] gh/rec/172/base -> origin/gh/rec/172/base 2025-12-04T09:20:54.5012984Z * [new branch] gh/rec/172/head -> origin/gh/rec/172/head 2025-12-04T09:20:54.5013127Z * [new branch] gh/rec/172/orig -> origin/gh/rec/172/orig 2025-12-04T09:20:54.5018845Z * [new branch] gh/rec/173/base -> origin/gh/rec/173/base 2025-12-04T09:20:54.5019034Z * [new branch] gh/rec/173/head -> origin/gh/rec/173/head 2025-12-04T09:20:54.5019174Z * [new branch] gh/rec/173/orig -> origin/gh/rec/173/orig 2025-12-04T09:20:54.5019305Z * [new branch] gh/rec/174/base -> origin/gh/rec/174/base 2025-12-04T09:20:54.5019437Z * [new branch] gh/rec/174/head -> origin/gh/rec/174/head 2025-12-04T09:20:54.5019565Z * [new branch] gh/rec/174/orig -> origin/gh/rec/174/orig 2025-12-04T09:20:54.5019869Z * [new branch] gh/rec/175/base -> origin/gh/rec/175/base 2025-12-04T09:20:54.5020008Z * [new branch] gh/rec/175/head -> origin/gh/rec/175/head 2025-12-04T09:20:54.5020138Z * [new branch] gh/rec/175/orig -> origin/gh/rec/175/orig 2025-12-04T09:20:54.5020757Z * [new branch] gh/rec/176/base -> origin/gh/rec/176/base 2025-12-04T09:20:54.5021339Z * [new branch] gh/rec/176/head -> origin/gh/rec/176/head 2025-12-04T09:20:54.5022155Z * [new branch] gh/rec/176/orig -> origin/gh/rec/176/orig 2025-12-04T09:20:54.5023503Z * [new branch] gh/rec/177/base -> origin/gh/rec/177/base 2025-12-04T09:20:54.5024069Z * [new branch] gh/rec/177/head -> origin/gh/rec/177/head 2025-12-04T09:20:54.5025040Z * [new branch] gh/rec/177/orig -> origin/gh/rec/177/orig 2025-12-04T09:20:54.5026250Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-12-04T09:20:54.5026848Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-12-04T09:20:54.5027492Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-12-04T09:20:54.5028689Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-12-04T09:20:54.5028894Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-12-04T09:20:54.5030678Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-12-04T09:20:54.5031307Z * [new branch] gh/robert-hardwick/5/base -> origin/gh/robert-hardwick/5/base 2025-12-04T09:20:54.5031540Z * [new branch] gh/robert-hardwick/5/head -> origin/gh/robert-hardwick/5/head 2025-12-04T09:20:54.5034684Z * [new branch] gh/robert-hardwick/5/orig -> origin/gh/robert-hardwick/5/orig 2025-12-04T09:20:54.5034993Z * [new branch] gh/robert-hardwick/6/base -> origin/gh/robert-hardwick/6/base 2025-12-04T09:20:54.5035593Z * [new branch] gh/robert-hardwick/6/head -> origin/gh/robert-hardwick/6/head 2025-12-04T09:20:54.5035787Z * [new branch] gh/robert-hardwick/6/orig -> origin/gh/robert-hardwick/6/orig 2025-12-04T09:20:54.5039757Z * [new branch] gh/robert-hardwick/7/base -> origin/gh/robert-hardwick/7/base 2025-12-04T09:20:54.5039926Z * [new branch] gh/robert-hardwick/7/head -> origin/gh/robert-hardwick/7/head 2025-12-04T09:20:54.5040084Z * [new branch] gh/robert-hardwick/7/orig -> origin/gh/robert-hardwick/7/orig 2025-12-04T09:20:54.5040253Z * [new branch] gh/robert-hardwick/8/base -> origin/gh/robert-hardwick/8/base 2025-12-04T09:20:54.5047384Z * [new branch] gh/robert-hardwick/8/head -> origin/gh/robert-hardwick/8/head 2025-12-04T09:20:54.5049460Z * [new branch] gh/robert-hardwick/8/orig -> origin/gh/robert-hardwick/8/orig 2025-12-04T09:20:54.5049646Z * [new branch] gh/robert-hardwick/9/base -> origin/gh/robert-hardwick/9/base 2025-12-04T09:20:54.5049978Z * [new branch] gh/robert-hardwick/9/head -> origin/gh/robert-hardwick/9/head 2025-12-04T09:20:54.5050161Z * [new branch] gh/robert-hardwick/9/orig -> origin/gh/robert-hardwick/9/orig 2025-12-04T09:20:54.5050325Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-12-04T09:20:54.5050475Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-12-04T09:20:54.5050612Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-12-04T09:20:54.5050754Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-12-04T09:20:54.5050903Z * [new branch] gh/rtimpe/22/base -> origin/gh/rtimpe/22/base 2025-12-04T09:20:54.5051201Z * [new branch] gh/rtimpe/22/head -> origin/gh/rtimpe/22/head 2025-12-04T09:20:54.5051355Z * [new branch] gh/rtimpe/22/orig -> origin/gh/rtimpe/22/orig 2025-12-04T09:20:54.5051720Z * [new branch] gh/rtimpe/23/base -> origin/gh/rtimpe/23/base 2025-12-04T09:20:54.5051961Z * [new branch] gh/rtimpe/23/head -> origin/gh/rtimpe/23/head 2025-12-04T09:20:54.5052112Z * [new branch] gh/rtimpe/23/orig -> origin/gh/rtimpe/23/orig 2025-12-04T09:20:54.5057531Z * [new branch] gh/rtimpe/24/base -> origin/gh/rtimpe/24/base 2025-12-04T09:20:54.5059556Z * [new branch] gh/rtimpe/24/head -> origin/gh/rtimpe/24/head 2025-12-04T09:20:54.5059743Z * [new branch] gh/rtimpe/24/orig -> origin/gh/rtimpe/24/orig 2025-12-04T09:20:54.5059888Z * [new branch] gh/rtimpe/25/base -> origin/gh/rtimpe/25/base 2025-12-04T09:20:54.5060053Z * [new branch] gh/rtimpe/25/head -> origin/gh/rtimpe/25/head 2025-12-04T09:20:54.5060197Z * [new branch] gh/rtimpe/25/orig -> origin/gh/rtimpe/25/orig 2025-12-04T09:20:54.5060338Z * [new branch] gh/rtimpe/26/base -> origin/gh/rtimpe/26/base 2025-12-04T09:20:54.5060489Z * [new branch] gh/rtimpe/26/head -> origin/gh/rtimpe/26/head 2025-12-04T09:20:54.5060629Z * [new branch] gh/rtimpe/26/orig -> origin/gh/rtimpe/26/orig 2025-12-04T09:20:54.5066009Z * [new branch] gh/rtimpe/27/base -> origin/gh/rtimpe/27/base 2025-12-04T09:20:54.5068207Z * [new branch] gh/rtimpe/27/head -> origin/gh/rtimpe/27/head 2025-12-04T09:20:54.5074103Z * [new branch] gh/rtimpe/27/orig -> origin/gh/rtimpe/27/orig 2025-12-04T09:20:54.5078799Z * [new branch] gh/rtimpe/28/base -> origin/gh/rtimpe/28/base 2025-12-04T09:20:54.5080874Z * [new branch] gh/rtimpe/28/head -> origin/gh/rtimpe/28/head 2025-12-04T09:20:54.5081150Z * [new branch] gh/rtimpe/28/orig -> origin/gh/rtimpe/28/orig 2025-12-04T09:20:54.5081684Z * [new branch] gh/rtimpe/29/base -> origin/gh/rtimpe/29/base 2025-12-04T09:20:54.5087704Z * [new branch] gh/rtimpe/29/head -> origin/gh/rtimpe/29/head 2025-12-04T09:20:54.5088015Z * [new branch] gh/rtimpe/29/orig -> origin/gh/rtimpe/29/orig 2025-12-04T09:20:54.5088193Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-12-04T09:20:54.5088356Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-12-04T09:20:54.5088583Z * [new branch] gh/rtimpe/30/base -> origin/gh/rtimpe/30/base 2025-12-04T09:20:54.5089283Z * [new branch] gh/rtimpe/30/head -> origin/gh/rtimpe/30/head 2025-12-04T09:20:54.5089701Z * [new branch] gh/rtimpe/30/orig -> origin/gh/rtimpe/30/orig 2025-12-04T09:20:54.5089858Z * [new branch] gh/rtimpe/31/base -> origin/gh/rtimpe/31/base 2025-12-04T09:20:54.5090008Z * [new branch] gh/rtimpe/31/head -> origin/gh/rtimpe/31/head 2025-12-04T09:20:54.5090190Z * [new branch] gh/rtimpe/31/orig -> origin/gh/rtimpe/31/orig 2025-12-04T09:20:54.5090327Z * [new branch] gh/rtimpe/32/base -> origin/gh/rtimpe/32/base 2025-12-04T09:20:54.5090473Z * [new branch] gh/rtimpe/32/head -> origin/gh/rtimpe/32/head 2025-12-04T09:20:54.5090616Z * [new branch] gh/rtimpe/32/orig -> origin/gh/rtimpe/32/orig 2025-12-04T09:20:54.5090759Z * [new branch] gh/rtimpe/33/base -> origin/gh/rtimpe/33/base 2025-12-04T09:20:54.5090895Z * [new branch] gh/rtimpe/33/head -> origin/gh/rtimpe/33/head 2025-12-04T09:20:54.5091270Z * [new branch] gh/rtimpe/33/orig -> origin/gh/rtimpe/33/orig 2025-12-04T09:20:54.5091413Z * [new branch] gh/rtimpe/34/base -> origin/gh/rtimpe/34/base 2025-12-04T09:20:54.5091782Z * [new branch] gh/rtimpe/34/head -> origin/gh/rtimpe/34/head 2025-12-04T09:20:54.5092009Z * [new branch] gh/rtimpe/34/orig -> origin/gh/rtimpe/34/orig 2025-12-04T09:20:54.5092162Z * [new branch] gh/rtimpe/35/base -> origin/gh/rtimpe/35/base 2025-12-04T09:20:54.5092295Z * [new branch] gh/rtimpe/35/head -> origin/gh/rtimpe/35/head 2025-12-04T09:20:54.5092427Z * [new branch] gh/rtimpe/35/orig -> origin/gh/rtimpe/35/orig 2025-12-04T09:20:54.5092573Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-12-04T09:20:54.5092715Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-12-04T09:20:54.5092892Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-12-04T09:20:54.5093044Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-12-04T09:20:54.5093194Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-12-04T09:20:54.5093354Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-12-04T09:20:54.5093498Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-12-04T09:20:54.5093650Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-12-04T09:20:54.5093793Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-12-04T09:20:54.5093934Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-12-04T09:20:54.5094084Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-12-04T09:20:54.5094231Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-12-04T09:20:54.5094380Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-12-04T09:20:54.5094527Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-12-04T09:20:54.5094668Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-12-04T09:20:54.5094818Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-12-04T09:20:54.5094964Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-12-04T09:20:54.5095109Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-12-04T09:20:54.5095650Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-12-04T09:20:54.5095791Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-12-04T09:20:54.5095934Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-12-04T09:20:54.5096070Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-12-04T09:20:54.5096219Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-12-04T09:20:54.5098044Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-12-04T09:20:54.5098390Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-12-04T09:20:54.5098579Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-12-04T09:20:54.5100401Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-12-04T09:20:54.5100741Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-12-04T09:20:54.5101181Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-12-04T09:20:54.5102846Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-12-04T09:20:54.5103208Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-12-04T09:20:54.5103545Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-12-04T09:20:54.5104074Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-12-04T09:20:54.5105353Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-12-04T09:20:54.5105671Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-12-04T09:20:54.5108013Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-12-04T09:20:54.5108367Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-12-04T09:20:54.5108542Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-12-04T09:20:54.5108761Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-12-04T09:20:54.5109758Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-12-04T09:20:54.5110148Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-12-04T09:20:54.5113590Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-12-04T09:20:54.5113771Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-12-04T09:20:54.5113912Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-12-04T09:20:54.5114057Z * [new branch] gh/seemethere/71/base -> origin/gh/seemethere/71/base 2025-12-04T09:20:54.5114213Z * [new branch] gh/seemethere/71/head -> origin/gh/seemethere/71/head 2025-12-04T09:20:54.5114501Z * [new branch] gh/seemethere/71/orig -> origin/gh/seemethere/71/orig 2025-12-04T09:20:54.5119139Z * [new branch] gh/seemethere/72/base -> origin/gh/seemethere/72/base 2025-12-04T09:20:54.5123743Z * [new branch] gh/seemethere/72/head -> origin/gh/seemethere/72/head 2025-12-04T09:20:54.5125838Z * [new branch] gh/seemethere/72/orig -> origin/gh/seemethere/72/orig 2025-12-04T09:20:54.5126139Z * [new branch] gh/seemethere/73/base -> origin/gh/seemethere/73/base 2025-12-04T09:20:54.5126509Z * [new branch] gh/seemethere/73/head -> origin/gh/seemethere/73/head 2025-12-04T09:20:54.5126685Z * [new branch] gh/seemethere/73/orig -> origin/gh/seemethere/73/orig 2025-12-04T09:20:54.5126851Z * [new branch] gh/seemethere/74/base -> origin/gh/seemethere/74/base 2025-12-04T09:20:54.5127028Z * [new branch] gh/seemethere/74/head -> origin/gh/seemethere/74/head 2025-12-04T09:20:54.5127178Z * [new branch] gh/seemethere/74/orig -> origin/gh/seemethere/74/orig 2025-12-04T09:20:54.5127326Z * [new branch] gh/seemethere/75/base -> origin/gh/seemethere/75/base 2025-12-04T09:20:54.5127479Z * [new branch] gh/seemethere/75/head -> origin/gh/seemethere/75/head 2025-12-04T09:20:54.5127630Z * [new branch] gh/seemethere/75/orig -> origin/gh/seemethere/75/orig 2025-12-04T09:20:54.5127777Z * [new branch] gh/seemethere/76/base -> origin/gh/seemethere/76/base 2025-12-04T09:20:54.5127918Z * [new branch] gh/seemethere/76/head -> origin/gh/seemethere/76/head 2025-12-04T09:20:54.5128071Z * [new branch] gh/seemethere/76/orig -> origin/gh/seemethere/76/orig 2025-12-04T09:20:54.5131039Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-12-04T09:20:54.5131293Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-12-04T09:20:54.5131770Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-12-04T09:20:54.5132092Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-12-04T09:20:54.5132477Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-12-04T09:20:54.5132644Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-12-04T09:20:54.5139845Z * [new branch] gh/shunting314/249/base -> origin/gh/shunting314/249/base 2025-12-04T09:20:54.5141823Z * [new branch] gh/shunting314/249/head -> origin/gh/shunting314/249/head 2025-12-04T09:20:54.5142454Z * [new branch] gh/shunting314/249/orig -> origin/gh/shunting314/249/orig 2025-12-04T09:20:54.5142719Z * [new branch] gh/shunting314/253/base -> origin/gh/shunting314/253/base 2025-12-04T09:20:54.5142909Z * [new branch] gh/shunting314/253/head -> origin/gh/shunting314/253/head 2025-12-04T09:20:54.5143067Z * [new branch] gh/shunting314/253/orig -> origin/gh/shunting314/253/orig 2025-12-04T09:20:54.5143232Z * [new branch] gh/shunting314/256/base -> origin/gh/shunting314/256/base 2025-12-04T09:20:54.5143392Z * [new branch] gh/shunting314/256/head -> origin/gh/shunting314/256/head 2025-12-04T09:20:54.5143546Z * [new branch] gh/shunting314/256/orig -> origin/gh/shunting314/256/orig 2025-12-04T09:20:54.5143698Z * [new branch] gh/shunting314/257/base -> origin/gh/shunting314/257/base 2025-12-04T09:20:54.5146638Z * [new branch] gh/shunting314/257/head -> origin/gh/shunting314/257/head 2025-12-04T09:20:54.5146820Z * [new branch] gh/shunting314/257/orig -> origin/gh/shunting314/257/orig 2025-12-04T09:20:54.5146986Z * [new branch] gh/shunting314/258/base -> origin/gh/shunting314/258/base 2025-12-04T09:20:54.5147146Z * [new branch] gh/shunting314/258/head -> origin/gh/shunting314/258/head 2025-12-04T09:20:54.5147304Z * [new branch] gh/shunting314/258/orig -> origin/gh/shunting314/258/orig 2025-12-04T09:20:54.5147469Z * [new branch] gh/shunting314/259/base -> origin/gh/shunting314/259/base 2025-12-04T09:20:54.5147625Z * [new branch] gh/shunting314/259/head -> origin/gh/shunting314/259/head 2025-12-04T09:20:54.5153390Z * [new branch] gh/shunting314/259/orig -> origin/gh/shunting314/259/orig 2025-12-04T09:20:54.5157444Z * [new branch] gh/shunting314/260/base -> origin/gh/shunting314/260/base 2025-12-04T09:20:54.5159514Z * [new branch] gh/shunting314/260/head -> origin/gh/shunting314/260/head 2025-12-04T09:20:54.5159858Z * [new branch] gh/shunting314/260/orig -> origin/gh/shunting314/260/orig 2025-12-04T09:20:54.5160165Z * [new branch] gh/shunting314/261/base -> origin/gh/shunting314/261/base 2025-12-04T09:20:54.5160506Z * [new branch] gh/shunting314/261/head -> origin/gh/shunting314/261/head 2025-12-04T09:20:54.5160781Z * [new branch] gh/shunting314/261/orig -> origin/gh/shunting314/261/orig 2025-12-04T09:20:54.5160920Z * [new branch] gh/shunting314/262/base -> origin/gh/shunting314/262/base 2025-12-04T09:20:54.5161054Z * [new branch] gh/shunting314/262/head -> origin/gh/shunting314/262/head 2025-12-04T09:20:54.5161197Z * [new branch] gh/shunting314/262/orig -> origin/gh/shunting314/262/orig 2025-12-04T09:20:54.5161331Z * [new branch] gh/shunting314/263/base -> origin/gh/shunting314/263/base 2025-12-04T09:20:54.5161623Z * [new branch] gh/shunting314/263/head -> origin/gh/shunting314/263/head 2025-12-04T09:20:54.5161759Z * [new branch] gh/shunting314/263/orig -> origin/gh/shunting314/263/orig 2025-12-04T09:20:54.5161895Z * [new branch] gh/shunting314/264/base -> origin/gh/shunting314/264/base 2025-12-04T09:20:54.5162100Z * [new branch] gh/shunting314/264/head -> origin/gh/shunting314/264/head 2025-12-04T09:20:54.5162239Z * [new branch] gh/shunting314/264/orig -> origin/gh/shunting314/264/orig 2025-12-04T09:20:54.5162376Z * [new branch] gh/shunting314/265/base -> origin/gh/shunting314/265/base 2025-12-04T09:20:54.5162519Z * [new branch] gh/shunting314/265/head -> origin/gh/shunting314/265/head 2025-12-04T09:20:54.5162656Z * [new branch] gh/shunting314/265/orig -> origin/gh/shunting314/265/orig 2025-12-04T09:20:54.5162799Z * [new branch] gh/shunting314/266/base -> origin/gh/shunting314/266/base 2025-12-04T09:20:54.5166705Z * [new branch] gh/shunting314/266/head -> origin/gh/shunting314/266/head 2025-12-04T09:20:54.5167118Z * [new branch] gh/shunting314/266/orig -> origin/gh/shunting314/266/orig 2025-12-04T09:20:54.5167320Z * [new branch] gh/shunting314/267/base -> origin/gh/shunting314/267/base 2025-12-04T09:20:54.5167469Z * [new branch] gh/shunting314/267/head -> origin/gh/shunting314/267/head 2025-12-04T09:20:54.5167752Z * [new branch] gh/shunting314/267/orig -> origin/gh/shunting314/267/orig 2025-12-04T09:20:54.5167942Z * [new branch] gh/shunting314/268/base -> origin/gh/shunting314/268/base 2025-12-04T09:20:54.5168111Z * [new branch] gh/shunting314/268/head -> origin/gh/shunting314/268/head 2025-12-04T09:20:54.5168788Z * [new branch] gh/shunting314/268/orig -> origin/gh/shunting314/268/orig 2025-12-04T09:20:54.5169142Z * [new branch] gh/shunting314/269/base -> origin/gh/shunting314/269/base 2025-12-04T09:20:54.5169372Z * [new branch] gh/shunting314/269/head -> origin/gh/shunting314/269/head 2025-12-04T09:20:54.5169563Z * [new branch] gh/shunting314/269/orig -> origin/gh/shunting314/269/orig 2025-12-04T09:20:54.5171517Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-12-04T09:20:54.5172526Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-12-04T09:20:54.5173022Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-12-04T09:20:54.5173791Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-12-04T09:20:54.5174739Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-12-04T09:20:54.5174937Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-12-04T09:20:54.5176277Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-12-04T09:20:54.5176727Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-12-04T09:20:54.5178650Z * [new branch] gh/slayton58/39/base -> origin/gh/slayton58/39/base 2025-12-04T09:20:54.5178860Z * [new branch] gh/slayton58/39/head -> origin/gh/slayton58/39/head 2025-12-04T09:20:54.5181206Z * [new branch] gh/slayton58/39/orig -> origin/gh/slayton58/39/orig 2025-12-04T09:20:54.5181387Z * [new branch] gh/slayton58/42/base -> origin/gh/slayton58/42/base 2025-12-04T09:20:54.5181549Z * [new branch] gh/slayton58/42/head -> origin/gh/slayton58/42/head 2025-12-04T09:20:54.5183017Z * [new branch] gh/slayton58/42/orig -> origin/gh/slayton58/42/orig 2025-12-04T09:20:54.5183203Z * [new branch] gh/slayton58/43/base -> origin/gh/slayton58/43/base 2025-12-04T09:20:54.5183688Z * [new branch] gh/slayton58/43/head -> origin/gh/slayton58/43/head 2025-12-04T09:20:54.5189406Z * [new branch] gh/slayton58/43/orig -> origin/gh/slayton58/43/orig 2025-12-04T09:20:54.5193153Z * [new branch] gh/slayton58/44/base -> origin/gh/slayton58/44/base 2025-12-04T09:20:54.5197635Z * [new branch] gh/slayton58/44/head -> origin/gh/slayton58/44/head 2025-12-04T09:20:54.5197985Z * [new branch] gh/slayton58/44/orig -> origin/gh/slayton58/44/orig 2025-12-04T09:20:54.5198314Z * [new branch] gh/slayton58/45/base -> origin/gh/slayton58/45/base 2025-12-04T09:20:54.5198472Z * [new branch] gh/slayton58/45/head -> origin/gh/slayton58/45/head 2025-12-04T09:20:54.5198611Z * [new branch] gh/slayton58/45/orig -> origin/gh/slayton58/45/orig 2025-12-04T09:20:54.5198749Z * [new branch] gh/slayton58/46/base -> origin/gh/slayton58/46/base 2025-12-04T09:20:54.5198904Z * [new branch] gh/slayton58/46/head -> origin/gh/slayton58/46/head 2025-12-04T09:20:54.5199057Z * [new branch] gh/slayton58/46/orig -> origin/gh/slayton58/46/orig 2025-12-04T09:20:54.5199218Z * [new branch] gh/slayton58/6/base -> origin/gh/slayton58/6/base 2025-12-04T09:20:54.5199365Z * [new branch] gh/slayton58/6/head -> origin/gh/slayton58/6/head 2025-12-04T09:20:54.5199503Z * [new branch] gh/slayton58/7/base -> origin/gh/slayton58/7/base 2025-12-04T09:20:54.5199647Z * [new branch] gh/slayton58/7/head -> origin/gh/slayton58/7/head 2025-12-04T09:20:54.5199800Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-12-04T09:20:54.5199952Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-12-04T09:20:54.5200092Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-12-04T09:20:54.5200234Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-12-04T09:20:54.5200381Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-12-04T09:20:54.5200527Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-12-04T09:20:54.5200767Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-12-04T09:20:54.5201543Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-12-04T09:20:54.5205980Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-12-04T09:20:54.5206321Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-12-04T09:20:54.5206500Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-12-04T09:20:54.5206668Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-12-04T09:20:54.5206956Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-12-04T09:20:54.5207189Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-12-04T09:20:54.5207558Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-12-04T09:20:54.5211731Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-12-04T09:20:54.5212113Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-12-04T09:20:54.5212300Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-12-04T09:20:54.5212497Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-12-04T09:20:54.5212661Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-12-04T09:20:54.5213492Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-12-04T09:20:54.5214137Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-12-04T09:20:54.5214958Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-12-04T09:20:54.5216040Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-12-04T09:20:54.5216384Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-12-04T09:20:54.5220274Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-12-04T09:20:54.5220454Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-12-04T09:20:54.5220599Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-12-04T09:20:54.5220752Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-12-04T09:20:54.5220928Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-12-04T09:20:54.5226385Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-12-04T09:20:54.5226716Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-12-04T09:20:54.5226916Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-12-04T09:20:54.5227175Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-12-04T09:20:54.5227384Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-12-04T09:20:54.5227538Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-12-04T09:20:54.5227805Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-12-04T09:20:54.5228003Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-12-04T09:20:54.5228282Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-12-04T09:20:54.5232651Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-12-04T09:20:54.5232905Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-12-04T09:20:54.5233070Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-12-04T09:20:54.5234621Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-12-04T09:20:54.5234795Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-12-04T09:20:54.5234948Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-12-04T09:20:54.5235091Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-12-04T09:20:54.5235453Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-12-04T09:20:54.5238264Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-12-04T09:20:54.5238601Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-12-04T09:20:54.5238793Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-12-04T09:20:54.5239062Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-12-04T09:20:54.5244695Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-12-04T09:20:54.5245040Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-12-04T09:20:54.5245214Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-12-04T09:20:54.5245358Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-12-04T09:20:54.5245833Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-12-04T09:20:54.5246118Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-12-04T09:20:54.5246262Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-12-04T09:20:54.5246473Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-12-04T09:20:54.5246621Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-12-04T09:20:54.5246773Z * [new branch] gh/soulitzer/380/base -> origin/gh/soulitzer/380/base 2025-12-04T09:20:54.5246917Z * [new branch] gh/soulitzer/380/head -> origin/gh/soulitzer/380/head 2025-12-04T09:20:54.5247059Z * [new branch] gh/soulitzer/380/orig -> origin/gh/soulitzer/380/orig 2025-12-04T09:20:54.5248471Z * [new branch] gh/soulitzer/385/base -> origin/gh/soulitzer/385/base 2025-12-04T09:20:54.5248968Z * [new branch] gh/soulitzer/385/head -> origin/gh/soulitzer/385/head 2025-12-04T09:20:54.5249517Z * [new branch] gh/soulitzer/385/orig -> origin/gh/soulitzer/385/orig 2025-12-04T09:20:54.5250743Z * [new branch] gh/soulitzer/386/base -> origin/gh/soulitzer/386/base 2025-12-04T09:20:54.5251067Z * [new branch] gh/soulitzer/386/head -> origin/gh/soulitzer/386/head 2025-12-04T09:20:54.5252403Z * [new branch] gh/soulitzer/386/orig -> origin/gh/soulitzer/386/orig 2025-12-04T09:20:54.5252984Z * [new branch] gh/soulitzer/387/base -> origin/gh/soulitzer/387/base 2025-12-04T09:20:54.5254001Z * [new branch] gh/soulitzer/387/head -> origin/gh/soulitzer/387/head 2025-12-04T09:20:54.5254196Z * [new branch] gh/soulitzer/387/orig -> origin/gh/soulitzer/387/orig 2025-12-04T09:20:54.5255746Z * [new branch] gh/soulitzer/388/base -> origin/gh/soulitzer/388/base 2025-12-04T09:20:54.5256046Z * [new branch] gh/soulitzer/388/head -> origin/gh/soulitzer/388/head 2025-12-04T09:20:54.5258296Z * [new branch] gh/soulitzer/388/orig -> origin/gh/soulitzer/388/orig 2025-12-04T09:20:54.5258506Z * [new branch] gh/soulitzer/389/base -> origin/gh/soulitzer/389/base 2025-12-04T09:20:54.5258822Z * [new branch] gh/soulitzer/389/head -> origin/gh/soulitzer/389/head 2025-12-04T09:20:54.5259691Z * [new branch] gh/soulitzer/389/orig -> origin/gh/soulitzer/389/orig 2025-12-04T09:20:54.5263707Z * [new branch] gh/soulitzer/390/base -> origin/gh/soulitzer/390/base 2025-12-04T09:20:54.5263897Z * [new branch] gh/soulitzer/390/head -> origin/gh/soulitzer/390/head 2025-12-04T09:20:54.5264040Z * [new branch] gh/soulitzer/390/orig -> origin/gh/soulitzer/390/orig 2025-12-04T09:20:54.5264199Z * [new branch] gh/soulitzer/391/base -> origin/gh/soulitzer/391/base 2025-12-04T09:20:54.5264335Z * [new branch] gh/soulitzer/391/head -> origin/gh/soulitzer/391/head 2025-12-04T09:20:54.5264471Z * [new branch] gh/soulitzer/391/orig -> origin/gh/soulitzer/391/orig 2025-12-04T09:20:54.5266050Z * [new branch] gh/soulitzer/392/base -> origin/gh/soulitzer/392/base 2025-12-04T09:20:54.5266427Z * [new branch] gh/soulitzer/392/head -> origin/gh/soulitzer/392/head 2025-12-04T09:20:54.5266604Z * [new branch] gh/soulitzer/392/orig -> origin/gh/soulitzer/392/orig 2025-12-04T09:20:54.5269075Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-12-04T09:20:54.5269392Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-12-04T09:20:54.5270997Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-12-04T09:20:54.5271296Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-12-04T09:20:54.5273469Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-12-04T09:20:54.5278563Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-12-04T09:20:54.5278755Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-12-04T09:20:54.5278899Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-12-04T09:20:54.5279041Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-12-04T09:20:54.5279180Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-12-04T09:20:54.5284601Z * [new branch] gh/swolchok/839/base -> origin/gh/swolchok/839/base 2025-12-04T09:20:54.5284808Z * [new branch] gh/swolchok/839/head -> origin/gh/swolchok/839/head 2025-12-04T09:20:54.5284956Z * [new branch] gh/swolchok/839/orig -> origin/gh/swolchok/839/orig 2025-12-04T09:20:54.5285107Z * [new branch] gh/swolchok/841/base -> origin/gh/swolchok/841/base 2025-12-04T09:20:54.5285255Z * [new branch] gh/swolchok/841/head -> origin/gh/swolchok/841/head 2025-12-04T09:20:54.5285403Z * [new branch] gh/swolchok/841/orig -> origin/gh/swolchok/841/orig 2025-12-04T09:20:54.5285829Z * [new branch] gh/swolchok/842/base -> origin/gh/swolchok/842/base 2025-12-04T09:20:54.5286579Z * [new branch] gh/swolchok/842/head -> origin/gh/swolchok/842/head 2025-12-04T09:20:54.5287262Z * [new branch] gh/swolchok/842/orig -> origin/gh/swolchok/842/orig 2025-12-04T09:20:54.5288405Z * [new branch] gh/swolchok/845/base -> origin/gh/swolchok/845/base 2025-12-04T09:20:54.5288675Z * [new branch] gh/swolchok/845/head -> origin/gh/swolchok/845/head 2025-12-04T09:20:54.5289886Z * [new branch] gh/swolchok/845/orig -> origin/gh/swolchok/845/orig 2025-12-04T09:20:54.5290984Z * [new branch] gh/swolchok/848/base -> origin/gh/swolchok/848/base 2025-12-04T09:20:54.5291620Z * [new branch] gh/swolchok/848/head -> origin/gh/swolchok/848/head 2025-12-04T09:20:54.5295825Z * [new branch] gh/swolchok/848/orig -> origin/gh/swolchok/848/orig 2025-12-04T09:20:54.5296296Z * [new branch] gh/swolchok/856/base -> origin/gh/swolchok/856/base 2025-12-04T09:20:54.5296443Z * [new branch] gh/swolchok/856/head -> origin/gh/swolchok/856/head 2025-12-04T09:20:54.5296588Z * [new branch] gh/swolchok/856/orig -> origin/gh/swolchok/856/orig 2025-12-04T09:20:54.5296725Z * [new branch] gh/swolchok/860/base -> origin/gh/swolchok/860/base 2025-12-04T09:20:54.5297023Z * [new branch] gh/swolchok/860/head -> origin/gh/swolchok/860/head 2025-12-04T09:20:54.5297246Z * [new branch] gh/swolchok/860/orig -> origin/gh/swolchok/860/orig 2025-12-04T09:20:54.5302008Z * [new branch] gh/swolchok/861/base -> origin/gh/swolchok/861/base 2025-12-04T09:20:54.5302484Z * [new branch] gh/swolchok/861/head -> origin/gh/swolchok/861/head 2025-12-04T09:20:54.5302646Z * [new branch] gh/swolchok/861/orig -> origin/gh/swolchok/861/orig 2025-12-04T09:20:54.5302957Z * [new branch] gh/swolchok/862/base -> origin/gh/swolchok/862/base 2025-12-04T09:20:54.5303133Z * [new branch] gh/swolchok/862/head -> origin/gh/swolchok/862/head 2025-12-04T09:20:54.5303304Z * [new branch] gh/swolchok/862/orig -> origin/gh/swolchok/862/orig 2025-12-04T09:20:54.5303688Z * [new branch] gh/swolchok/863/base -> origin/gh/swolchok/863/base 2025-12-04T09:20:54.5304322Z * [new branch] gh/swolchok/863/head -> origin/gh/swolchok/863/head 2025-12-04T09:20:54.5305466Z * [new branch] gh/swolchok/863/orig -> origin/gh/swolchok/863/orig 2025-12-04T09:20:54.5306425Z * [new branch] gh/swolchok/864/base -> origin/gh/swolchok/864/base 2025-12-04T09:20:54.5306741Z * [new branch] gh/swolchok/864/head -> origin/gh/swolchok/864/head 2025-12-04T09:20:54.5307807Z * [new branch] gh/swolchok/864/orig -> origin/gh/swolchok/864/orig 2025-12-04T09:20:54.5308908Z * [new branch] gh/swolchok/865/base -> origin/gh/swolchok/865/base 2025-12-04T09:20:54.5309757Z * [new branch] gh/swolchok/865/head -> origin/gh/swolchok/865/head 2025-12-04T09:20:54.5310234Z * [new branch] gh/swolchok/865/orig -> origin/gh/swolchok/865/orig 2025-12-04T09:20:54.5312115Z * [new branch] gh/swolchok/866/base -> origin/gh/swolchok/866/base 2025-12-04T09:20:54.5312589Z * [new branch] gh/swolchok/866/head -> origin/gh/swolchok/866/head 2025-12-04T09:20:54.5313456Z * [new branch] gh/swolchok/866/orig -> origin/gh/swolchok/866/orig 2025-12-04T09:20:54.5314903Z * [new branch] gh/swolchok/867/base -> origin/gh/swolchok/867/base 2025-12-04T09:20:54.5315151Z * [new branch] gh/swolchok/867/head -> origin/gh/swolchok/867/head 2025-12-04T09:20:54.5316276Z * [new branch] gh/swolchok/867/orig -> origin/gh/swolchok/867/orig 2025-12-04T09:20:54.5317293Z * [new branch] gh/swolchok/868/base -> origin/gh/swolchok/868/base 2025-12-04T09:20:54.5317687Z * [new branch] gh/swolchok/868/head -> origin/gh/swolchok/868/head 2025-12-04T09:20:54.5318752Z * [new branch] gh/swolchok/868/orig -> origin/gh/swolchok/868/orig 2025-12-04T09:20:54.5319834Z * [new branch] gh/swolchok/869/base -> origin/gh/swolchok/869/base 2025-12-04T09:20:54.5320098Z * [new branch] gh/swolchok/869/head -> origin/gh/swolchok/869/head 2025-12-04T09:20:54.5321162Z * [new branch] gh/swolchok/869/orig -> origin/gh/swolchok/869/orig 2025-12-04T09:20:54.5322301Z * [new branch] gh/swolchok/870/base -> origin/gh/swolchok/870/base 2025-12-04T09:20:54.5322489Z * [new branch] gh/swolchok/870/head -> origin/gh/swolchok/870/head 2025-12-04T09:20:54.5323605Z * [new branch] gh/swolchok/870/orig -> origin/gh/swolchok/870/orig 2025-12-04T09:20:54.5326894Z * [new branch] gh/swolchok/871/base -> origin/gh/swolchok/871/base 2025-12-04T09:20:54.5327506Z * [new branch] gh/swolchok/871/head -> origin/gh/swolchok/871/head 2025-12-04T09:20:54.5328129Z * [new branch] gh/swolchok/871/orig -> origin/gh/swolchok/871/orig 2025-12-04T09:20:54.5328482Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-12-04T09:20:54.5328643Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-12-04T09:20:54.5328800Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-12-04T09:20:54.5329981Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-12-04T09:20:54.5330387Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-12-04T09:20:54.5331708Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-12-04T09:20:54.5336120Z * [new branch] gh/tianyu-l/3/base -> origin/gh/tianyu-l/3/base 2025-12-04T09:20:54.5336310Z * [new branch] gh/tianyu-l/3/orig -> origin/gh/tianyu-l/3/orig 2025-12-04T09:20:54.5336507Z * [new branch] gh/tianyu-l/4/base -> origin/gh/tianyu-l/4/base 2025-12-04T09:20:54.5336945Z * [new branch] gh/tianyu-l/4/head -> origin/gh/tianyu-l/4/head 2025-12-04T09:20:54.5337105Z * [new branch] gh/tianyu-l/4/orig -> origin/gh/tianyu-l/4/orig 2025-12-04T09:20:54.5342146Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-12-04T09:20:54.5342399Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-12-04T09:20:54.5342634Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-12-04T09:20:54.5343236Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-12-04T09:20:54.5343464Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-12-04T09:20:54.5343640Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-12-04T09:20:54.5346416Z * [new branch] gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base 2025-12-04T09:20:54.5346629Z * [new branch] gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head 2025-12-04T09:20:54.5346796Z * [new branch] gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig 2025-12-04T09:20:54.5346991Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-12-04T09:20:54.5347160Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-12-04T09:20:54.5347312Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-12-04T09:20:54.5352108Z * [new branch] gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base 2025-12-04T09:20:54.5352313Z * [new branch] gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head 2025-12-04T09:20:54.5352478Z * [new branch] gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig 2025-12-04T09:20:54.5352666Z * [new branch] gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base 2025-12-04T09:20:54.5352824Z * [new branch] gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head 2025-12-04T09:20:54.5354835Z * [new branch] gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig 2025-12-04T09:20:54.5355226Z * [new branch] gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base 2025-12-04T09:20:54.5355428Z * [new branch] gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head 2025-12-04T09:20:54.5355716Z * [new branch] gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig 2025-12-04T09:20:54.5356091Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2025-12-04T09:20:54.5356276Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2025-12-04T09:20:54.5358549Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2025-12-04T09:20:54.5358745Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2025-12-04T09:20:54.5358908Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2025-12-04T09:20:54.5359107Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2025-12-04T09:20:54.5359272Z * [new branch] gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base 2025-12-04T09:20:54.5363610Z * [new branch] gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head 2025-12-04T09:20:54.5363816Z * [new branch] gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig 2025-12-04T09:20:54.5364153Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2025-12-04T09:20:54.5364525Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2025-12-04T09:20:54.5364711Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2025-12-04T09:20:54.5364924Z * [new branch] gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base 2025-12-04T09:20:54.5365311Z * [new branch] gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head 2025-12-04T09:20:54.5365943Z * [new branch] gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig 2025-12-04T09:20:54.5366170Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2025-12-04T09:20:54.5366349Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2025-12-04T09:20:54.5366525Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2025-12-04T09:20:54.5369453Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2025-12-04T09:20:54.5369683Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2025-12-04T09:20:54.5369862Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2025-12-04T09:20:54.5370040Z * [new branch] gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base 2025-12-04T09:20:54.5370208Z * [new branch] gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head 2025-12-04T09:20:54.5370382Z * [new branch] gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig 2025-12-04T09:20:54.5372157Z * [new branch] gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base 2025-12-04T09:20:54.5372519Z * [new branch] gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head 2025-12-04T09:20:54.5372847Z * [new branch] gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig 2025-12-04T09:20:54.5376495Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-12-04T09:20:54.5376699Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-12-04T09:20:54.5376881Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-12-04T09:20:54.5377143Z * [new branch] gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base 2025-12-04T09:20:54.5377476Z * [new branch] gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head 2025-12-04T09:20:54.5380874Z * [new branch] gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig 2025-12-04T09:20:54.5381125Z * [new branch] gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base 2025-12-04T09:20:54.5381423Z * [new branch] gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head 2025-12-04T09:20:54.5381612Z * [new branch] gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig 2025-12-04T09:20:54.5381851Z * [new branch] gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base 2025-12-04T09:20:54.5382038Z * [new branch] gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head 2025-12-04T09:20:54.5387610Z * [new branch] gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig 2025-12-04T09:20:54.5392441Z * [new branch] gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base 2025-12-04T09:20:54.5396429Z * [new branch] gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head 2025-12-04T09:20:54.5401834Z * [new branch] gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig 2025-12-04T09:20:54.5405894Z * [new branch] gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base 2025-12-04T09:20:54.5406107Z * [new branch] gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head 2025-12-04T09:20:54.5406499Z * [new branch] gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig 2025-12-04T09:20:54.5406681Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-12-04T09:20:54.5406852Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-12-04T09:20:54.5407086Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-12-04T09:20:54.5407248Z * [new branch] gh/tugsbayasgalan/70/base -> origin/gh/tugsbayasgalan/70/base 2025-12-04T09:20:54.5407401Z * [new branch] gh/tugsbayasgalan/70/head -> origin/gh/tugsbayasgalan/70/head 2025-12-04T09:20:54.5407562Z * [new branch] gh/tugsbayasgalan/70/orig -> origin/gh/tugsbayasgalan/70/orig 2025-12-04T09:20:54.5407712Z * [new branch] gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base 2025-12-04T09:20:54.5407867Z * [new branch] gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head 2025-12-04T09:20:54.5408026Z * [new branch] gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig 2025-12-04T09:20:54.5408177Z * [new branch] gh/tugsbayasgalan/72/base -> origin/gh/tugsbayasgalan/72/base 2025-12-04T09:20:54.5408349Z * [new branch] gh/tugsbayasgalan/72/head -> origin/gh/tugsbayasgalan/72/head 2025-12-04T09:20:54.5408501Z * [new branch] gh/tugsbayasgalan/72/orig -> origin/gh/tugsbayasgalan/72/orig 2025-12-04T09:20:54.5408650Z * [new branch] gh/tugsbayasgalan/73/base -> origin/gh/tugsbayasgalan/73/base 2025-12-04T09:20:54.5408809Z * [new branch] gh/tugsbayasgalan/73/head -> origin/gh/tugsbayasgalan/73/head 2025-12-04T09:20:54.5408960Z * [new branch] gh/tugsbayasgalan/73/orig -> origin/gh/tugsbayasgalan/73/orig 2025-12-04T09:20:54.5409119Z * [new branch] gh/tugsbayasgalan/74/base -> origin/gh/tugsbayasgalan/74/base 2025-12-04T09:20:54.5409271Z * [new branch] gh/tugsbayasgalan/74/head -> origin/gh/tugsbayasgalan/74/head 2025-12-04T09:20:54.5409421Z * [new branch] gh/tugsbayasgalan/74/orig -> origin/gh/tugsbayasgalan/74/orig 2025-12-04T09:20:54.5409589Z * [new branch] gh/tugsbayasgalan/75/base -> origin/gh/tugsbayasgalan/75/base 2025-12-04T09:20:54.5409741Z * [new branch] gh/tugsbayasgalan/75/head -> origin/gh/tugsbayasgalan/75/head 2025-12-04T09:20:54.5409899Z * [new branch] gh/tugsbayasgalan/75/orig -> origin/gh/tugsbayasgalan/75/orig 2025-12-04T09:20:54.5410049Z * [new branch] gh/tugsbayasgalan/76/base -> origin/gh/tugsbayasgalan/76/base 2025-12-04T09:20:54.5410198Z * [new branch] gh/tugsbayasgalan/76/head -> origin/gh/tugsbayasgalan/76/head 2025-12-04T09:20:54.5410356Z * [new branch] gh/tugsbayasgalan/76/orig -> origin/gh/tugsbayasgalan/76/orig 2025-12-04T09:20:54.5410516Z * [new branch] gh/tugsbayasgalan/77/base -> origin/gh/tugsbayasgalan/77/base 2025-12-04T09:20:54.5410680Z * [new branch] gh/tugsbayasgalan/77/head -> origin/gh/tugsbayasgalan/77/head 2025-12-04T09:20:54.5411193Z * [new branch] gh/tugsbayasgalan/77/orig -> origin/gh/tugsbayasgalan/77/orig 2025-12-04T09:20:54.5416517Z * [new branch] gh/tugsbayasgalan/78/base -> origin/gh/tugsbayasgalan/78/base 2025-12-04T09:20:54.5416729Z * [new branch] gh/tugsbayasgalan/78/head -> origin/gh/tugsbayasgalan/78/head 2025-12-04T09:20:54.5416891Z * [new branch] gh/tugsbayasgalan/78/orig -> origin/gh/tugsbayasgalan/78/orig 2025-12-04T09:20:54.5417058Z * [new branch] gh/tugsbayasgalan/79/base -> origin/gh/tugsbayasgalan/79/base 2025-12-04T09:20:54.5417211Z * [new branch] gh/tugsbayasgalan/79/head -> origin/gh/tugsbayasgalan/79/head 2025-12-04T09:20:54.5417515Z * [new branch] gh/tugsbayasgalan/79/orig -> origin/gh/tugsbayasgalan/79/orig 2025-12-04T09:20:54.5417718Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-12-04T09:20:54.5418299Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-12-04T09:20:54.5419033Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-12-04T09:20:54.5422921Z * [new branch] gh/tugsbayasgalan/80/base -> origin/gh/tugsbayasgalan/80/base 2025-12-04T09:20:54.5423135Z * [new branch] gh/tugsbayasgalan/80/head -> origin/gh/tugsbayasgalan/80/head 2025-12-04T09:20:54.5423303Z * [new branch] gh/tugsbayasgalan/80/orig -> origin/gh/tugsbayasgalan/80/orig 2025-12-04T09:20:54.5423483Z * [new branch] gh/tugsbayasgalan/81/base -> origin/gh/tugsbayasgalan/81/base 2025-12-04T09:20:54.5423655Z * [new branch] gh/tugsbayasgalan/81/head -> origin/gh/tugsbayasgalan/81/head 2025-12-04T09:20:54.5423991Z * [new branch] gh/tugsbayasgalan/81/orig -> origin/gh/tugsbayasgalan/81/orig 2025-12-04T09:20:54.5425594Z * [new branch] gh/tugsbayasgalan/82/base -> origin/gh/tugsbayasgalan/82/base 2025-12-04T09:20:54.5425980Z * [new branch] gh/tugsbayasgalan/82/head -> origin/gh/tugsbayasgalan/82/head 2025-12-04T09:20:54.5430293Z * [new branch] gh/tugsbayasgalan/82/orig -> origin/gh/tugsbayasgalan/82/orig 2025-12-04T09:20:54.5430494Z * [new branch] gh/tugsbayasgalan/83/base -> origin/gh/tugsbayasgalan/83/base 2025-12-04T09:20:54.5430682Z * [new branch] gh/tugsbayasgalan/83/head -> origin/gh/tugsbayasgalan/83/head 2025-12-04T09:20:54.5430875Z * [new branch] gh/tugsbayasgalan/83/orig -> origin/gh/tugsbayasgalan/83/orig 2025-12-04T09:20:54.5431047Z * [new branch] gh/tugsbayasgalan/84/base -> origin/gh/tugsbayasgalan/84/base 2025-12-04T09:20:54.5431569Z * [new branch] gh/tugsbayasgalan/84/head -> origin/gh/tugsbayasgalan/84/head 2025-12-04T09:20:54.5432169Z * [new branch] gh/tugsbayasgalan/84/orig -> origin/gh/tugsbayasgalan/84/orig 2025-12-04T09:20:54.5433671Z * [new branch] gh/tugsbayasgalan/85/base -> origin/gh/tugsbayasgalan/85/base 2025-12-04T09:20:54.5433871Z * [new branch] gh/tugsbayasgalan/85/head -> origin/gh/tugsbayasgalan/85/head 2025-12-04T09:20:54.5434487Z * [new branch] gh/tugsbayasgalan/85/orig -> origin/gh/tugsbayasgalan/85/orig 2025-12-04T09:20:54.5438779Z * [new branch] gh/tugsbayasgalan/86/base -> origin/gh/tugsbayasgalan/86/base 2025-12-04T09:20:54.5438973Z * [new branch] gh/tugsbayasgalan/86/head -> origin/gh/tugsbayasgalan/86/head 2025-12-04T09:20:54.5439142Z * [new branch] gh/tugsbayasgalan/86/orig -> origin/gh/tugsbayasgalan/86/orig 2025-12-04T09:20:54.5439297Z * [new branch] gh/tugsbayasgalan/87/base -> origin/gh/tugsbayasgalan/87/base 2025-12-04T09:20:54.5439464Z * [new branch] gh/tugsbayasgalan/87/head -> origin/gh/tugsbayasgalan/87/head 2025-12-04T09:20:54.5440561Z * [new branch] gh/tugsbayasgalan/87/orig -> origin/gh/tugsbayasgalan/87/orig 2025-12-04T09:20:54.5441623Z * [new branch] gh/tugsbayasgalan/88/base -> origin/gh/tugsbayasgalan/88/base 2025-12-04T09:20:54.5442114Z * [new branch] gh/tugsbayasgalan/88/head -> origin/gh/tugsbayasgalan/88/head 2025-12-04T09:20:54.5442735Z * [new branch] gh/tugsbayasgalan/88/orig -> origin/gh/tugsbayasgalan/88/orig 2025-12-04T09:20:54.5443968Z * [new branch] gh/tugsbayasgalan/89/base -> origin/gh/tugsbayasgalan/89/base 2025-12-04T09:20:54.5444702Z * [new branch] gh/tugsbayasgalan/89/head -> origin/gh/tugsbayasgalan/89/head 2025-12-04T09:20:54.5445411Z * [new branch] gh/tugsbayasgalan/89/orig -> origin/gh/tugsbayasgalan/89/orig 2025-12-04T09:20:54.5446406Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-12-04T09:20:54.5446903Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-12-04T09:20:54.5448177Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-12-04T09:20:54.5449016Z * [new branch] gh/tugsbayasgalan/90/base -> origin/gh/tugsbayasgalan/90/base 2025-12-04T09:20:54.5449515Z * [new branch] gh/tugsbayasgalan/90/head -> origin/gh/tugsbayasgalan/90/head 2025-12-04T09:20:54.5450566Z * [new branch] gh/tugsbayasgalan/90/orig -> origin/gh/tugsbayasgalan/90/orig 2025-12-04T09:20:54.5452053Z * [new branch] gh/tugsbayasgalan/91/base -> origin/gh/tugsbayasgalan/91/base 2025-12-04T09:20:54.5452351Z * [new branch] gh/tugsbayasgalan/91/head -> origin/gh/tugsbayasgalan/91/head 2025-12-04T09:20:54.5454330Z * [new branch] gh/tugsbayasgalan/91/orig -> origin/gh/tugsbayasgalan/91/orig 2025-12-04T09:20:54.5454570Z * [new branch] gh/tugsbayasgalan/92/base -> origin/gh/tugsbayasgalan/92/base 2025-12-04T09:20:54.5454756Z * [new branch] gh/tugsbayasgalan/92/head -> origin/gh/tugsbayasgalan/92/head 2025-12-04T09:20:54.5455592Z * [new branch] gh/tugsbayasgalan/92/orig -> origin/gh/tugsbayasgalan/92/orig 2025-12-04T09:20:54.5456818Z * [new branch] gh/tugsbayasgalan/93/base -> origin/gh/tugsbayasgalan/93/base 2025-12-04T09:20:54.5457226Z * [new branch] gh/tugsbayasgalan/93/head -> origin/gh/tugsbayasgalan/93/head 2025-12-04T09:20:54.5458291Z * [new branch] gh/tugsbayasgalan/93/orig -> origin/gh/tugsbayasgalan/93/orig 2025-12-04T09:20:54.5459556Z * [new branch] gh/v0i0/14/base -> origin/gh/v0i0/14/base 2025-12-04T09:20:54.5460048Z * [new branch] gh/v0i0/14/head -> origin/gh/v0i0/14/head 2025-12-04T09:20:54.5461295Z * [new branch] gh/v0i0/14/orig -> origin/gh/v0i0/14/orig 2025-12-04T09:20:54.5461676Z * [new branch] gh/v0i0/15/base -> origin/gh/v0i0/15/base 2025-12-04T09:20:54.5462659Z * [new branch] gh/v0i0/15/head -> origin/gh/v0i0/15/head 2025-12-04T09:20:54.5463124Z * [new branch] gh/v0i0/15/orig -> origin/gh/v0i0/15/orig 2025-12-04T09:20:54.5464391Z * [new branch] gh/v0i0/16/base -> origin/gh/v0i0/16/base 2025-12-04T09:20:54.5464771Z * [new branch] gh/v0i0/16/head -> origin/gh/v0i0/16/head 2025-12-04T09:20:54.5465722Z * [new branch] gh/v0i0/16/orig -> origin/gh/v0i0/16/orig 2025-12-04T09:20:54.5466695Z * [new branch] gh/v0i0/17/base -> origin/gh/v0i0/17/base 2025-12-04T09:20:54.5467170Z * [new branch] gh/v0i0/17/head -> origin/gh/v0i0/17/head 2025-12-04T09:20:54.5469018Z * [new branch] gh/v0i0/17/orig -> origin/gh/v0i0/17/orig 2025-12-04T09:20:54.5469276Z * [new branch] gh/v0i0/18/base -> origin/gh/v0i0/18/base 2025-12-04T09:20:54.5469640Z * [new branch] gh/v0i0/18/head -> origin/gh/v0i0/18/head 2025-12-04T09:20:54.5470796Z * [new branch] gh/v0i0/18/orig -> origin/gh/v0i0/18/orig 2025-12-04T09:20:54.5471470Z * [new branch] gh/v0i0/19/base -> origin/gh/v0i0/19/base 2025-12-04T09:20:54.5472969Z * [new branch] gh/v0i0/19/head -> origin/gh/v0i0/19/head 2025-12-04T09:20:54.5473424Z * [new branch] gh/v0i0/19/orig -> origin/gh/v0i0/19/orig 2025-12-04T09:20:54.5474389Z * [new branch] gh/vishal9-team/1/base -> origin/gh/vishal9-team/1/base 2025-12-04T09:20:54.5474898Z * [new branch] gh/vishal9-team/1/head -> origin/gh/vishal9-team/1/head 2025-12-04T09:20:54.5475709Z * [new branch] gh/vishal9-team/2/base -> origin/gh/vishal9-team/2/base 2025-12-04T09:20:54.5476619Z * [new branch] gh/vishal9-team/2/head -> origin/gh/vishal9-team/2/head 2025-12-04T09:20:54.5477515Z * [new branch] gh/vishal9-team/2/orig -> origin/gh/vishal9-team/2/orig 2025-12-04T09:20:54.5478613Z * [new branch] gh/vishal9-team/3/base -> origin/gh/vishal9-team/3/base 2025-12-04T09:20:54.5481199Z * [new branch] gh/vishal9-team/3/head -> origin/gh/vishal9-team/3/head 2025-12-04T09:20:54.5481363Z * [new branch] gh/vishal9-team/3/orig -> origin/gh/vishal9-team/3/orig 2025-12-04T09:20:54.5481514Z * [new branch] gh/vishal9-team/4/base -> origin/gh/vishal9-team/4/base 2025-12-04T09:20:54.5481884Z * [new branch] gh/vishal9-team/4/head -> origin/gh/vishal9-team/4/head 2025-12-04T09:20:54.5486206Z * [new branch] gh/vishal9-team/4/orig -> origin/gh/vishal9-team/4/orig 2025-12-04T09:20:54.5486378Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-12-04T09:20:54.5486528Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-12-04T09:20:54.5486659Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-12-04T09:20:54.5491118Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-12-04T09:20:54.5491322Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-12-04T09:20:54.5491841Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-12-04T09:20:54.5492007Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-12-04T09:20:54.5492163Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-12-04T09:20:54.5492318Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-12-04T09:20:54.5495695Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-12-04T09:20:54.5495893Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-12-04T09:20:54.5496070Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-12-04T09:20:54.5496220Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-12-04T09:20:54.5496441Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-12-04T09:20:54.5496589Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-12-04T09:20:54.5503733Z * [new branch] gh/wconstab/448/base -> origin/gh/wconstab/448/base 2025-12-04T09:20:54.5506509Z * [new branch] gh/wconstab/448/head -> origin/gh/wconstab/448/head 2025-12-04T09:20:54.5506765Z * [new branch] gh/wconstab/448/orig -> origin/gh/wconstab/448/orig 2025-12-04T09:20:54.5506901Z * [new branch] gh/wconstab/449/base -> origin/gh/wconstab/449/base 2025-12-04T09:20:54.5507030Z * [new branch] gh/wconstab/449/head -> origin/gh/wconstab/449/head 2025-12-04T09:20:54.5507174Z * [new branch] gh/wconstab/449/orig -> origin/gh/wconstab/449/orig 2025-12-04T09:20:54.5507314Z * [new branch] gh/wconstab/450/base -> origin/gh/wconstab/450/base 2025-12-04T09:20:54.5507448Z * [new branch] gh/wconstab/450/head -> origin/gh/wconstab/450/head 2025-12-04T09:20:54.5507576Z * [new branch] gh/wconstab/450/orig -> origin/gh/wconstab/450/orig 2025-12-04T09:20:54.5507721Z * [new branch] gh/wconstab/451/base -> origin/gh/wconstab/451/base 2025-12-04T09:20:54.5507902Z * [new branch] gh/wconstab/451/head -> origin/gh/wconstab/451/head 2025-12-04T09:20:54.5508209Z * [new branch] gh/wconstab/451/orig -> origin/gh/wconstab/451/orig 2025-12-04T09:20:54.5508359Z * [new branch] gh/wconstab/452/base -> origin/gh/wconstab/452/base 2025-12-04T09:20:54.5508496Z * [new branch] gh/wconstab/452/head -> origin/gh/wconstab/452/head 2025-12-04T09:20:54.5508699Z * [new branch] gh/wconstab/452/orig -> origin/gh/wconstab/452/orig 2025-12-04T09:20:54.5511542Z * [new branch] gh/wconstab/453/base -> origin/gh/wconstab/453/base 2025-12-04T09:20:54.5511758Z * [new branch] gh/wconstab/453/head -> origin/gh/wconstab/453/head 2025-12-04T09:20:54.5511923Z * [new branch] gh/wconstab/453/orig -> origin/gh/wconstab/453/orig 2025-12-04T09:20:54.5512053Z * [new branch] gh/wconstab/454/base -> origin/gh/wconstab/454/base 2025-12-04T09:20:54.5512182Z * [new branch] gh/wconstab/454/head -> origin/gh/wconstab/454/head 2025-12-04T09:20:54.5512473Z * [new branch] gh/wconstab/454/orig -> origin/gh/wconstab/454/orig 2025-12-04T09:20:54.5512662Z * [new branch] gh/wconstab/455/base -> origin/gh/wconstab/455/base 2025-12-04T09:20:54.5512812Z * [new branch] gh/wconstab/455/head -> origin/gh/wconstab/455/head 2025-12-04T09:20:54.5513090Z * [new branch] gh/wconstab/455/orig -> origin/gh/wconstab/455/orig 2025-12-04T09:20:54.5517508Z * [new branch] gh/wconstab/456/base -> origin/gh/wconstab/456/base 2025-12-04T09:20:54.5517680Z * [new branch] gh/wconstab/456/head -> origin/gh/wconstab/456/head 2025-12-04T09:20:54.5517815Z * [new branch] gh/wconstab/456/orig -> origin/gh/wconstab/456/orig 2025-12-04T09:20:54.5517960Z * [new branch] gh/wconstab/457/base -> origin/gh/wconstab/457/base 2025-12-04T09:20:54.5518098Z * [new branch] gh/wconstab/457/head -> origin/gh/wconstab/457/head 2025-12-04T09:20:54.5518340Z * [new branch] gh/wconstab/457/orig -> origin/gh/wconstab/457/orig 2025-12-04T09:20:54.5519975Z * [new branch] gh/wconstab/458/base -> origin/gh/wconstab/458/base 2025-12-04T09:20:54.5520612Z * [new branch] gh/wconstab/458/head -> origin/gh/wconstab/458/head 2025-12-04T09:20:54.5520859Z * [new branch] gh/wconstab/458/orig -> origin/gh/wconstab/458/orig 2025-12-04T09:20:54.5526924Z * [new branch] gh/wconstab/459/base -> origin/gh/wconstab/459/base 2025-12-04T09:20:54.5527269Z * [new branch] gh/wconstab/459/head -> origin/gh/wconstab/459/head 2025-12-04T09:20:54.5527445Z * [new branch] gh/wconstab/459/orig -> origin/gh/wconstab/459/orig 2025-12-04T09:20:54.5527672Z * [new branch] gh/wconstab/460/base -> origin/gh/wconstab/460/base 2025-12-04T09:20:54.5527943Z * [new branch] gh/wconstab/460/head -> origin/gh/wconstab/460/head 2025-12-04T09:20:54.5528121Z * [new branch] gh/wconstab/460/orig -> origin/gh/wconstab/460/orig 2025-12-04T09:20:54.5528777Z * [new branch] gh/wconstab/461/base -> origin/gh/wconstab/461/base 2025-12-04T09:20:54.5528989Z * [new branch] gh/wconstab/461/head -> origin/gh/wconstab/461/head 2025-12-04T09:20:54.5529142Z * [new branch] gh/wconstab/461/orig -> origin/gh/wconstab/461/orig 2025-12-04T09:20:54.5529459Z * [new branch] gh/wconstab/462/base -> origin/gh/wconstab/462/base 2025-12-04T09:20:54.5531172Z * [new branch] gh/wconstab/462/head -> origin/gh/wconstab/462/head 2025-12-04T09:20:54.5531718Z * [new branch] gh/wconstab/462/orig -> origin/gh/wconstab/462/orig 2025-12-04T09:20:54.5532205Z * [new branch] gh/wconstab/463/base -> origin/gh/wconstab/463/base 2025-12-04T09:20:54.5534645Z * [new branch] gh/wconstab/463/head -> origin/gh/wconstab/463/head 2025-12-04T09:20:54.5534989Z * [new branch] gh/wconstab/463/orig -> origin/gh/wconstab/463/orig 2025-12-04T09:20:54.5535196Z * [new branch] gh/wconstab/464/base -> origin/gh/wconstab/464/base 2025-12-04T09:20:54.5535732Z * [new branch] gh/wconstab/464/head -> origin/gh/wconstab/464/head 2025-12-04T09:20:54.5540717Z * [new branch] gh/wconstab/464/orig -> origin/gh/wconstab/464/orig 2025-12-04T09:20:54.5540896Z * [new branch] gh/wconstab/465/base -> origin/gh/wconstab/465/base 2025-12-04T09:20:54.5541041Z * [new branch] gh/wconstab/465/head -> origin/gh/wconstab/465/head 2025-12-04T09:20:54.5543606Z * [new branch] gh/wconstab/465/orig -> origin/gh/wconstab/465/orig 2025-12-04T09:20:54.5543928Z * [new branch] gh/wconstab/466/base -> origin/gh/wconstab/466/base 2025-12-04T09:20:54.5544181Z * [new branch] gh/wconstab/466/head -> origin/gh/wconstab/466/head 2025-12-04T09:20:54.5544348Z * [new branch] gh/wconstab/466/orig -> origin/gh/wconstab/466/orig 2025-12-04T09:20:54.5544581Z * [new branch] gh/wconstab/467/base -> origin/gh/wconstab/467/base 2025-12-04T09:20:54.5544880Z * [new branch] gh/wconstab/467/head -> origin/gh/wconstab/467/head 2025-12-04T09:20:54.5545511Z * [new branch] gh/wconstab/467/orig -> origin/gh/wconstab/467/orig 2025-12-04T09:20:54.5545677Z * [new branch] gh/wconstab/468/base -> origin/gh/wconstab/468/base 2025-12-04T09:20:54.5546043Z * [new branch] gh/wconstab/468/head -> origin/gh/wconstab/468/head 2025-12-04T09:20:54.5546203Z * [new branch] gh/wconstab/468/orig -> origin/gh/wconstab/468/orig 2025-12-04T09:20:54.5546640Z * [new branch] gh/weifengpy/39/base -> origin/gh/weifengpy/39/base 2025-12-04T09:20:54.5547327Z * [new branch] gh/weifengpy/39/head -> origin/gh/weifengpy/39/head 2025-12-04T09:20:54.5550645Z * [new branch] gh/weifengpy/39/orig -> origin/gh/weifengpy/39/orig 2025-12-04T09:20:54.5550970Z * [new branch] gh/weifengpy/40/base -> origin/gh/weifengpy/40/base 2025-12-04T09:20:54.5551215Z * [new branch] gh/weifengpy/40/head -> origin/gh/weifengpy/40/head 2025-12-04T09:20:54.5551422Z * [new branch] gh/weifengpy/40/orig -> origin/gh/weifengpy/40/orig 2025-12-04T09:20:54.5551566Z * [new branch] gh/weifengpy/41/base -> origin/gh/weifengpy/41/base 2025-12-04T09:20:54.5551714Z * [new branch] gh/weifengpy/41/head -> origin/gh/weifengpy/41/head 2025-12-04T09:20:54.5553343Z * [new branch] gh/weifengpy/41/orig -> origin/gh/weifengpy/41/orig 2025-12-04T09:20:54.5553829Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-12-04T09:20:54.5556170Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-12-04T09:20:54.5556370Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-12-04T09:20:54.5556530Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-12-04T09:20:54.5557210Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-12-04T09:20:54.5557802Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-12-04T09:20:54.5562066Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-12-04T09:20:54.5562564Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-12-04T09:20:54.5562722Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-12-04T09:20:54.5563047Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-12-04T09:20:54.5563190Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-12-04T09:20:54.5563328Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-12-04T09:20:54.5563551Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-12-04T09:20:54.5563964Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-12-04T09:20:54.5565809Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-12-04T09:20:54.5566001Z * [new branch] gh/williamwen42/296/base -> origin/gh/williamwen42/296/base 2025-12-04T09:20:54.5566505Z * [new branch] gh/williamwen42/296/head -> origin/gh/williamwen42/296/head 2025-12-04T09:20:54.5567351Z * [new branch] gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig 2025-12-04T09:20:54.5568389Z * [new branch] gh/williamwen42/297/base -> origin/gh/williamwen42/297/base 2025-12-04T09:20:54.5568678Z * [new branch] gh/williamwen42/297/head -> origin/gh/williamwen42/297/head 2025-12-04T09:20:54.5569817Z * [new branch] gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig 2025-12-04T09:20:54.5570853Z * [new branch] gh/williamwen42/306/base -> origin/gh/williamwen42/306/base 2025-12-04T09:20:54.5571340Z * [new branch] gh/williamwen42/306/head -> origin/gh/williamwen42/306/head 2025-12-04T09:20:54.5572774Z * [new branch] gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig 2025-12-04T09:20:54.5578604Z * [new branch] gh/williamwen42/309/base -> origin/gh/williamwen42/309/base 2025-12-04T09:20:54.5579160Z * [new branch] gh/williamwen42/309/head -> origin/gh/williamwen42/309/head 2025-12-04T09:20:54.5580116Z * [new branch] gh/williamwen42/309/orig -> origin/gh/williamwen42/309/orig 2025-12-04T09:20:54.5581442Z * [new branch] gh/williamwen42/310/base -> origin/gh/williamwen42/310/base 2025-12-04T09:20:54.5581783Z * [new branch] gh/williamwen42/310/head -> origin/gh/williamwen42/310/head 2025-12-04T09:20:54.5582212Z * [new branch] gh/williamwen42/310/orig -> origin/gh/williamwen42/310/orig 2025-12-04T09:20:54.5584601Z * [new branch] gh/williamwen42/311/base -> origin/gh/williamwen42/311/base 2025-12-04T09:20:54.5585812Z * [new branch] gh/williamwen42/311/head -> origin/gh/williamwen42/311/head 2025-12-04T09:20:54.5586053Z * [new branch] gh/williamwen42/311/orig -> origin/gh/williamwen42/311/orig 2025-12-04T09:20:54.5590138Z * [new branch] gh/williamwen42/319/base -> origin/gh/williamwen42/319/base 2025-12-04T09:20:54.5590467Z * [new branch] gh/williamwen42/319/head -> origin/gh/williamwen42/319/head 2025-12-04T09:20:54.5590637Z * [new branch] gh/williamwen42/319/orig -> origin/gh/williamwen42/319/orig 2025-12-04T09:20:54.5590786Z * [new branch] gh/williamwen42/325/base -> origin/gh/williamwen42/325/base 2025-12-04T09:20:54.5590944Z * [new branch] gh/williamwen42/325/head -> origin/gh/williamwen42/325/head 2025-12-04T09:20:54.5591096Z * [new branch] gh/williamwen42/325/orig -> origin/gh/williamwen42/325/orig 2025-12-04T09:20:54.5592718Z * [new branch] gh/williamwen42/326/base -> origin/gh/williamwen42/326/base 2025-12-04T09:20:54.5592925Z * [new branch] gh/williamwen42/326/head -> origin/gh/williamwen42/326/head 2025-12-04T09:20:54.5593379Z * [new branch] gh/williamwen42/326/orig -> origin/gh/williamwen42/326/orig 2025-12-04T09:20:54.5595228Z * [new branch] gh/williamwen42/327/base -> origin/gh/williamwen42/327/base 2025-12-04T09:20:54.5595731Z * [new branch] gh/williamwen42/327/head -> origin/gh/williamwen42/327/head 2025-12-04T09:20:54.5595899Z * [new branch] gh/williamwen42/327/orig -> origin/gh/williamwen42/327/orig 2025-12-04T09:20:54.5597600Z * [new branch] gh/williamwen42/328/base -> origin/gh/williamwen42/328/base 2025-12-04T09:20:54.5598037Z * [new branch] gh/williamwen42/328/head -> origin/gh/williamwen42/328/head 2025-12-04T09:20:54.5598247Z * [new branch] gh/williamwen42/328/orig -> origin/gh/williamwen42/328/orig 2025-12-04T09:20:54.5604143Z * [new branch] gh/williamwen42/329/base -> origin/gh/williamwen42/329/base 2025-12-04T09:20:54.5604522Z * [new branch] gh/williamwen42/329/head -> origin/gh/williamwen42/329/head 2025-12-04T09:20:54.5604789Z * [new branch] gh/williamwen42/329/orig -> origin/gh/williamwen42/329/orig 2025-12-04T09:20:54.5604981Z * [new branch] gh/williamwen42/330/base -> origin/gh/williamwen42/330/base 2025-12-04T09:20:54.5605256Z * [new branch] gh/williamwen42/330/head -> origin/gh/williamwen42/330/head 2025-12-04T09:20:54.5605446Z * [new branch] gh/williamwen42/330/orig -> origin/gh/williamwen42/330/orig 2025-12-04T09:20:54.5605727Z * [new branch] gh/williamwen42/331/base -> origin/gh/williamwen42/331/base 2025-12-04T09:20:54.5606411Z * [new branch] gh/williamwen42/331/head -> origin/gh/williamwen42/331/head 2025-12-04T09:20:54.5608323Z * [new branch] gh/williamwen42/331/orig -> origin/gh/williamwen42/331/orig 2025-12-04T09:20:54.5608530Z * [new branch] gh/williamwen42/332/base -> origin/gh/williamwen42/332/base 2025-12-04T09:20:54.5608719Z * [new branch] gh/williamwen42/332/head -> origin/gh/williamwen42/332/head 2025-12-04T09:20:54.5608898Z * [new branch] gh/williamwen42/332/orig -> origin/gh/williamwen42/332/orig 2025-12-04T09:20:54.5614878Z * [new branch] gh/williamwen42/333/base -> origin/gh/williamwen42/333/base 2025-12-04T09:20:54.5615078Z * [new branch] gh/williamwen42/333/head -> origin/gh/williamwen42/333/head 2025-12-04T09:20:54.5615263Z * [new branch] gh/williamwen42/333/orig -> origin/gh/williamwen42/333/orig 2025-12-04T09:20:54.5615419Z * [new branch] gh/williamwen42/334/base -> origin/gh/williamwen42/334/base 2025-12-04T09:20:54.5615576Z * [new branch] gh/williamwen42/334/head -> origin/gh/williamwen42/334/head 2025-12-04T09:20:54.5615742Z * [new branch] gh/williamwen42/334/orig -> origin/gh/williamwen42/334/orig 2025-12-04T09:20:54.5617935Z * [new branch] gh/williamwen42/335/base -> origin/gh/williamwen42/335/base 2025-12-04T09:20:54.5618290Z * [new branch] gh/williamwen42/335/head -> origin/gh/williamwen42/335/head 2025-12-04T09:20:54.5618756Z * [new branch] gh/williamwen42/335/orig -> origin/gh/williamwen42/335/orig 2025-12-04T09:20:54.5620912Z * [new branch] gh/williamwen42/336/base -> origin/gh/williamwen42/336/base 2025-12-04T09:20:54.5621269Z * [new branch] gh/williamwen42/336/head -> origin/gh/williamwen42/336/head 2025-12-04T09:20:54.5623367Z * [new branch] gh/williamwen42/336/orig -> origin/gh/williamwen42/336/orig 2025-12-04T09:20:54.5623725Z * [new branch] gh/williamwen42/337/base -> origin/gh/williamwen42/337/base 2025-12-04T09:20:54.5623899Z * [new branch] gh/williamwen42/337/head -> origin/gh/williamwen42/337/head 2025-12-04T09:20:54.5624048Z * [new branch] gh/williamwen42/337/orig -> origin/gh/williamwen42/337/orig 2025-12-04T09:20:54.5626337Z * [new branch] gh/williamwen42/338/base -> origin/gh/williamwen42/338/base 2025-12-04T09:20:54.5626688Z * [new branch] gh/williamwen42/338/head -> origin/gh/williamwen42/338/head 2025-12-04T09:20:54.5627143Z * [new branch] gh/williamwen42/338/orig -> origin/gh/williamwen42/338/orig 2025-12-04T09:20:54.5627313Z * [new branch] gh/williamwen42/339/base -> origin/gh/williamwen42/339/base 2025-12-04T09:20:54.5628409Z * [new branch] gh/williamwen42/339/head -> origin/gh/williamwen42/339/head 2025-12-04T09:20:54.5628708Z * [new branch] gh/williamwen42/339/orig -> origin/gh/williamwen42/339/orig 2025-12-04T09:20:54.5634592Z * [new branch] gh/williamwen42/340/base -> origin/gh/williamwen42/340/base 2025-12-04T09:20:54.5637194Z * [new branch] gh/williamwen42/340/head -> origin/gh/williamwen42/340/head 2025-12-04T09:20:54.5641641Z * [new branch] gh/williamwen42/340/orig -> origin/gh/williamwen42/340/orig 2025-12-04T09:20:54.5644188Z * [new branch] gh/williamwen42/341/base -> origin/gh/williamwen42/341/base 2025-12-04T09:20:54.5644416Z * [new branch] gh/williamwen42/341/head -> origin/gh/williamwen42/341/head 2025-12-04T09:20:54.5644600Z * [new branch] gh/williamwen42/341/orig -> origin/gh/williamwen42/341/orig 2025-12-04T09:20:54.5644777Z * [new branch] gh/williamwen42/342/base -> origin/gh/williamwen42/342/base 2025-12-04T09:20:54.5644945Z * [new branch] gh/williamwen42/342/head -> origin/gh/williamwen42/342/head 2025-12-04T09:20:54.5645097Z * [new branch] gh/williamwen42/342/orig -> origin/gh/williamwen42/342/orig 2025-12-04T09:20:54.5645262Z * [new branch] gh/williamwen42/343/base -> origin/gh/williamwen42/343/base 2025-12-04T09:20:54.5645548Z * [new branch] gh/williamwen42/343/head -> origin/gh/williamwen42/343/head 2025-12-04T09:20:54.5645787Z * [new branch] gh/williamwen42/343/orig -> origin/gh/williamwen42/343/orig 2025-12-04T09:20:54.5645941Z * [new branch] gh/williamwen42/344/base -> origin/gh/williamwen42/344/base 2025-12-04T09:20:54.5646102Z * [new branch] gh/williamwen42/344/head -> origin/gh/williamwen42/344/head 2025-12-04T09:20:54.5646563Z * [new branch] gh/williamwen42/344/orig -> origin/gh/williamwen42/344/orig 2025-12-04T09:20:54.5646778Z * [new branch] gh/williamwen42/345/base -> origin/gh/williamwen42/345/base 2025-12-04T09:20:54.5646949Z * [new branch] gh/williamwen42/345/head -> origin/gh/williamwen42/345/head 2025-12-04T09:20:54.5647106Z * [new branch] gh/williamwen42/345/orig -> origin/gh/williamwen42/345/orig 2025-12-04T09:20:54.5647263Z * [new branch] gh/williamwen42/346/base -> origin/gh/williamwen42/346/base 2025-12-04T09:20:54.5647426Z * [new branch] gh/williamwen42/346/head -> origin/gh/williamwen42/346/head 2025-12-04T09:20:54.5647588Z * [new branch] gh/williamwen42/346/orig -> origin/gh/williamwen42/346/orig 2025-12-04T09:20:54.5647778Z * [new branch] gh/williamwen42/347/base -> origin/gh/williamwen42/347/base 2025-12-04T09:20:54.5648368Z * [new branch] gh/williamwen42/347/head -> origin/gh/williamwen42/347/head 2025-12-04T09:20:54.5649027Z * [new branch] gh/williamwen42/347/orig -> origin/gh/williamwen42/347/orig 2025-12-04T09:20:54.5653197Z * [new branch] gh/williamwen42/348/base -> origin/gh/williamwen42/348/base 2025-12-04T09:20:54.5658396Z * [new branch] gh/williamwen42/348/head -> origin/gh/williamwen42/348/head 2025-12-04T09:20:54.5662261Z * [new branch] gh/williamwen42/348/orig -> origin/gh/williamwen42/348/orig 2025-12-04T09:20:54.5667791Z * [new branch] gh/williamwen42/349/base -> origin/gh/williamwen42/349/base 2025-12-04T09:20:54.5671904Z * [new branch] gh/williamwen42/349/head -> origin/gh/williamwen42/349/head 2025-12-04T09:20:54.5674180Z * [new branch] gh/williamwen42/349/orig -> origin/gh/williamwen42/349/orig 2025-12-04T09:20:54.5674480Z * [new branch] gh/williamwen42/350/base -> origin/gh/williamwen42/350/base 2025-12-04T09:20:54.5678551Z * [new branch] gh/williamwen42/350/head -> origin/gh/williamwen42/350/head 2025-12-04T09:20:54.5679051Z * [new branch] gh/williamwen42/350/orig -> origin/gh/williamwen42/350/orig 2025-12-04T09:20:54.5684464Z * [new branch] gh/williamwen42/351/base -> origin/gh/williamwen42/351/base 2025-12-04T09:20:54.5689289Z * [new branch] gh/williamwen42/351/head -> origin/gh/williamwen42/351/head 2025-12-04T09:20:54.5689477Z * [new branch] gh/williamwen42/351/orig -> origin/gh/williamwen42/351/orig 2025-12-04T09:20:54.5689830Z * [new branch] gh/williamwen42/352/base -> origin/gh/williamwen42/352/base 2025-12-04T09:20:54.5690011Z * [new branch] gh/williamwen42/352/head -> origin/gh/williamwen42/352/head 2025-12-04T09:20:54.5690170Z * [new branch] gh/williamwen42/352/orig -> origin/gh/williamwen42/352/orig 2025-12-04T09:20:54.5690327Z * [new branch] gh/williamwen42/353/base -> origin/gh/williamwen42/353/base 2025-12-04T09:20:54.5690472Z * [new branch] gh/williamwen42/353/head -> origin/gh/williamwen42/353/head 2025-12-04T09:20:54.5690623Z * [new branch] gh/williamwen42/353/orig -> origin/gh/williamwen42/353/orig 2025-12-04T09:20:54.5690778Z * [new branch] gh/williamwen42/354/base -> origin/gh/williamwen42/354/base 2025-12-04T09:20:54.5690921Z * [new branch] gh/williamwen42/354/head -> origin/gh/williamwen42/354/head 2025-12-04T09:20:54.5691071Z * [new branch] gh/williamwen42/354/orig -> origin/gh/williamwen42/354/orig 2025-12-04T09:20:54.5691228Z * [new branch] gh/williamwen42/355/base -> origin/gh/williamwen42/355/base 2025-12-04T09:20:54.5691381Z * [new branch] gh/williamwen42/355/head -> origin/gh/williamwen42/355/head 2025-12-04T09:20:54.5691681Z * [new branch] gh/williamwen42/355/orig -> origin/gh/williamwen42/355/orig 2025-12-04T09:20:54.5691849Z * [new branch] gh/williamwen42/356/base -> origin/gh/williamwen42/356/base 2025-12-04T09:20:54.5691997Z * [new branch] gh/williamwen42/356/head -> origin/gh/williamwen42/356/head 2025-12-04T09:20:54.5692152Z * [new branch] gh/williamwen42/356/orig -> origin/gh/williamwen42/356/orig 2025-12-04T09:20:54.5692297Z * [new branch] gh/williamwen42/357/base -> origin/gh/williamwen42/357/base 2025-12-04T09:20:54.5692450Z * [new branch] gh/williamwen42/357/head -> origin/gh/williamwen42/357/head 2025-12-04T09:20:54.5692597Z * [new branch] gh/williamwen42/357/orig -> origin/gh/williamwen42/357/orig 2025-12-04T09:20:54.5692754Z * [new branch] gh/williamwen42/358/base -> origin/gh/williamwen42/358/base 2025-12-04T09:20:54.5692910Z * [new branch] gh/williamwen42/358/head -> origin/gh/williamwen42/358/head 2025-12-04T09:20:54.5693057Z * [new branch] gh/williamwen42/358/orig -> origin/gh/williamwen42/358/orig 2025-12-04T09:20:54.5693212Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-12-04T09:20:54.5693349Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-12-04T09:20:54.5693479Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-12-04T09:20:54.5693618Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-12-04T09:20:54.5693746Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-12-04T09:20:54.5693880Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-12-04T09:20:54.5694008Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-12-04T09:20:54.5694236Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-12-04T09:20:54.5694375Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-12-04T09:20:54.5694504Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-12-04T09:20:54.5694677Z * [new branch] gh/xmfan/301/base -> origin/gh/xmfan/301/base 2025-12-04T09:20:54.5694818Z * [new branch] gh/xmfan/301/head -> origin/gh/xmfan/301/head 2025-12-04T09:20:54.5694944Z * [new branch] gh/xmfan/301/orig -> origin/gh/xmfan/301/orig 2025-12-04T09:20:54.5695080Z * [new branch] gh/xmfan/304/base -> origin/gh/xmfan/304/base 2025-12-04T09:20:54.5695208Z * [new branch] gh/xmfan/304/head -> origin/gh/xmfan/304/head 2025-12-04T09:20:54.5695334Z * [new branch] gh/xmfan/304/orig -> origin/gh/xmfan/304/orig 2025-12-04T09:20:54.5695475Z * [new branch] gh/xmfan/309/base -> origin/gh/xmfan/309/base 2025-12-04T09:20:54.5695601Z * [new branch] gh/xmfan/309/head -> origin/gh/xmfan/309/head 2025-12-04T09:20:54.5695736Z * [new branch] gh/xmfan/309/orig -> origin/gh/xmfan/309/orig 2025-12-04T09:20:54.5695870Z * [new branch] gh/xmfan/310/base -> origin/gh/xmfan/310/base 2025-12-04T09:20:54.5695997Z * [new branch] gh/xmfan/310/head -> origin/gh/xmfan/310/head 2025-12-04T09:20:54.5698630Z * [new branch] gh/xmfan/310/orig -> origin/gh/xmfan/310/orig 2025-12-04T09:20:54.5699279Z * [new branch] gh/xmfan/311/base -> origin/gh/xmfan/311/base 2025-12-04T09:20:54.5703276Z * [new branch] gh/xmfan/311/head -> origin/gh/xmfan/311/head 2025-12-04T09:20:54.5703517Z * [new branch] gh/xmfan/311/orig -> origin/gh/xmfan/311/orig 2025-12-04T09:20:54.5703684Z * [new branch] gh/xmfan/312/base -> origin/gh/xmfan/312/base 2025-12-04T09:20:54.5703918Z * [new branch] gh/xmfan/312/head -> origin/gh/xmfan/312/head 2025-12-04T09:20:54.5704295Z * [new branch] gh/xmfan/312/orig -> origin/gh/xmfan/312/orig 2025-12-04T09:20:54.5704439Z * [new branch] gh/xmfan/313/base -> origin/gh/xmfan/313/base 2025-12-04T09:20:54.5704570Z * [new branch] gh/xmfan/313/head -> origin/gh/xmfan/313/head 2025-12-04T09:20:54.5704713Z * [new branch] gh/xmfan/313/orig -> origin/gh/xmfan/313/orig 2025-12-04T09:20:54.5704954Z * [new branch] gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base 2025-12-04T09:20:54.5708103Z * [new branch] gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head 2025-12-04T09:20:54.5708310Z * [new branch] gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig 2025-12-04T09:20:54.5708505Z * [new branch] gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base 2025-12-04T09:20:54.5708672Z * [new branch] gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head 2025-12-04T09:20:54.5709030Z * [new branch] gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig 2025-12-04T09:20:54.5709207Z * [new branch] gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base 2025-12-04T09:20:54.5709789Z * [new branch] gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head 2025-12-04T09:20:54.5715414Z * [new branch] gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig 2025-12-04T09:20:54.5717522Z * [new branch] gh/xuanzhang816/34/base -> origin/gh/xuanzhang816/34/base 2025-12-04T09:20:54.5717721Z * [new branch] gh/xuanzhang816/34/head -> origin/gh/xuanzhang816/34/head 2025-12-04T09:20:54.5718030Z * [new branch] gh/xuanzhang816/34/orig -> origin/gh/xuanzhang816/34/orig 2025-12-04T09:20:54.5718186Z * [new branch] gh/xuanzhang816/35/base -> origin/gh/xuanzhang816/35/base 2025-12-04T09:20:54.5718662Z * [new branch] gh/xuanzhang816/35/head -> origin/gh/xuanzhang816/35/head 2025-12-04T09:20:54.5719173Z * [new branch] gh/xuanzhang816/35/orig -> origin/gh/xuanzhang816/35/orig 2025-12-04T09:20:54.5719330Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-12-04T09:20:54.5719467Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-12-04T09:20:54.5719611Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-12-04T09:20:54.5722391Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-12-04T09:20:54.5722738Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-12-04T09:20:54.5722914Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-12-04T09:20:54.5723052Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-12-04T09:20:54.5723179Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-12-04T09:20:54.5723328Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-12-04T09:20:54.5723460Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-12-04T09:20:54.5723589Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-12-04T09:20:54.5727304Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-12-04T09:20:54.5727456Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-12-04T09:20:54.5727604Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-12-04T09:20:54.5727764Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-12-04T09:20:54.5727907Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-12-04T09:20:54.5728050Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-12-04T09:20:54.5728184Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-12-04T09:20:54.5728327Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-12-04T09:20:54.5728469Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-12-04T09:20:54.5728708Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-12-04T09:20:54.5730642Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-12-04T09:20:54.5730859Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-12-04T09:20:54.5731007Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-12-04T09:20:54.5732094Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-12-04T09:20:54.5737085Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-12-04T09:20:54.5741249Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-12-04T09:20:54.5746109Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-12-04T09:20:54.5750485Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-12-04T09:20:54.5750907Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-12-04T09:20:54.5751045Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-12-04T09:20:54.5751343Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-12-04T09:20:54.5751475Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-12-04T09:20:54.5751609Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-12-04T09:20:54.5751802Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-12-04T09:20:54.5751931Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-12-04T09:20:54.5752059Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-12-04T09:20:54.5752195Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-12-04T09:20:54.5752329Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-12-04T09:20:54.5752470Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-12-04T09:20:54.5752624Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-12-04T09:20:54.5752785Z * [new branch] gh/yang-yu-hang/1/base -> origin/gh/yang-yu-hang/1/base 2025-12-04T09:20:54.5752938Z * [new branch] gh/yang-yu-hang/1/head -> origin/gh/yang-yu-hang/1/head 2025-12-04T09:20:54.5753083Z * [new branch] gh/yang-yu-hang/1/orig -> origin/gh/yang-yu-hang/1/orig 2025-12-04T09:20:54.5753230Z * [new branch] gh/yang-yu-hang/2/base -> origin/gh/yang-yu-hang/2/base 2025-12-04T09:20:54.5753366Z * [new branch] gh/yang-yu-hang/2/head -> origin/gh/yang-yu-hang/2/head 2025-12-04T09:20:54.5753505Z * [new branch] gh/yang-yu-hang/2/orig -> origin/gh/yang-yu-hang/2/orig 2025-12-04T09:20:54.5753647Z * [new branch] gh/yang-yu-hang/3/base -> origin/gh/yang-yu-hang/3/base 2025-12-04T09:20:54.5753787Z * [new branch] gh/yang-yu-hang/3/head -> origin/gh/yang-yu-hang/3/head 2025-12-04T09:20:54.5753933Z * [new branch] gh/yang-yu-hang/3/orig -> origin/gh/yang-yu-hang/3/orig 2025-12-04T09:20:54.5755430Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-12-04T09:20:54.5755613Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-12-04T09:20:54.5761657Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-12-04T09:20:54.5766533Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-12-04T09:20:54.5768622Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-12-04T09:20:54.5768813Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-12-04T09:20:54.5768964Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-12-04T09:20:54.5769115Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-12-04T09:20:54.5769262Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-12-04T09:20:54.5769405Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-12-04T09:20:54.5769547Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-12-04T09:20:54.5769690Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-12-04T09:20:54.5769824Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-12-04T09:20:54.5769966Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-12-04T09:20:54.5770103Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-12-04T09:20:54.5770242Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-12-04T09:20:54.5770563Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-12-04T09:20:54.5770707Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-12-04T09:20:54.5770848Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-12-04T09:20:54.5771025Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-12-04T09:20:54.5771161Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-12-04T09:20:54.5771311Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-12-04T09:20:54.5771638Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-12-04T09:20:54.5771802Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-12-04T09:20:54.5772851Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-12-04T09:20:54.5773083Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-12-04T09:20:54.5776259Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-12-04T09:20:54.5776612Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-12-04T09:20:54.5776883Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-12-04T09:20:54.5777044Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-12-04T09:20:54.5777326Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-12-04T09:20:54.5778253Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-12-04T09:20:54.5778646Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-12-04T09:20:54.5781116Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-12-04T09:20:54.5781307Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-12-04T09:20:54.5781461Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-12-04T09:20:54.5785791Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-12-04T09:20:54.5785992Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-12-04T09:20:54.5786128Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-12-04T09:20:54.5786266Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-12-04T09:20:54.5786398Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-12-04T09:20:54.5786521Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-12-04T09:20:54.5787002Z * [new branch] gh/ydwu4/327/base -> origin/gh/ydwu4/327/base 2025-12-04T09:20:54.5787740Z * [new branch] gh/ydwu4/327/head -> origin/gh/ydwu4/327/head 2025-12-04T09:20:54.5788288Z * [new branch] gh/ydwu4/327/orig -> origin/gh/ydwu4/327/orig 2025-12-04T09:20:54.5792321Z * [new branch] gh/ydwu4/328/base -> origin/gh/ydwu4/328/base 2025-12-04T09:20:54.5792502Z * [new branch] gh/ydwu4/328/head -> origin/gh/ydwu4/328/head 2025-12-04T09:20:54.5792820Z * [new branch] gh/ydwu4/328/orig -> origin/gh/ydwu4/328/orig 2025-12-04T09:20:54.5792974Z * [new branch] gh/ydwu4/329/base -> origin/gh/ydwu4/329/base 2025-12-04T09:20:54.5793098Z * [new branch] gh/ydwu4/329/head -> origin/gh/ydwu4/329/head 2025-12-04T09:20:54.5793235Z * [new branch] gh/ydwu4/329/orig -> origin/gh/ydwu4/329/orig 2025-12-04T09:20:54.5795024Z * [new branch] gh/ydwu4/330/base -> origin/gh/ydwu4/330/base 2025-12-04T09:20:54.5795440Z * [new branch] gh/ydwu4/330/head -> origin/gh/ydwu4/330/head 2025-12-04T09:20:54.5795598Z * [new branch] gh/ydwu4/330/orig -> origin/gh/ydwu4/330/orig 2025-12-04T09:20:54.5796107Z * [new branch] gh/ydwu4/331/base -> origin/gh/ydwu4/331/base 2025-12-04T09:20:54.5797619Z * [new branch] gh/ydwu4/331/head -> origin/gh/ydwu4/331/head 2025-12-04T09:20:54.5797967Z * [new branch] gh/ydwu4/331/orig -> origin/gh/ydwu4/331/orig 2025-12-04T09:20:54.5798404Z * [new branch] gh/ydwu4/332/base -> origin/gh/ydwu4/332/base 2025-12-04T09:20:54.5800910Z * [new branch] gh/ydwu4/332/head -> origin/gh/ydwu4/332/head 2025-12-04T09:20:54.5801248Z * [new branch] gh/ydwu4/332/orig -> origin/gh/ydwu4/332/orig 2025-12-04T09:20:54.5801430Z * [new branch] gh/ydwu4/333/base -> origin/gh/ydwu4/333/base 2025-12-04T09:20:54.5801599Z * [new branch] gh/ydwu4/333/head -> origin/gh/ydwu4/333/head 2025-12-04T09:20:54.5802735Z * [new branch] gh/ydwu4/333/orig -> origin/gh/ydwu4/333/orig 2025-12-04T09:20:54.5803168Z * [new branch] gh/ydwu4/334/base -> origin/gh/ydwu4/334/base 2025-12-04T09:20:54.5804250Z * [new branch] gh/ydwu4/334/head -> origin/gh/ydwu4/334/head 2025-12-04T09:20:54.5804583Z * [new branch] gh/ydwu4/334/orig -> origin/gh/ydwu4/334/orig 2025-12-04T09:20:54.5805583Z * [new branch] gh/ydwu4/335/base -> origin/gh/ydwu4/335/base 2025-12-04T09:20:54.5805905Z * [new branch] gh/ydwu4/335/head -> origin/gh/ydwu4/335/head 2025-12-04T09:20:54.5806984Z * [new branch] gh/ydwu4/335/orig -> origin/gh/ydwu4/335/orig 2025-12-04T09:20:54.5808177Z * [new branch] gh/ydwu4/337/base -> origin/gh/ydwu4/337/base 2025-12-04T09:20:54.5808590Z * [new branch] gh/ydwu4/337/head -> origin/gh/ydwu4/337/head 2025-12-04T09:20:54.5809540Z * [new branch] gh/ydwu4/337/orig -> origin/gh/ydwu4/337/orig 2025-12-04T09:20:54.5810561Z * [new branch] gh/ydwu4/339/base -> origin/gh/ydwu4/339/base 2025-12-04T09:20:54.5811006Z * [new branch] gh/ydwu4/339/head -> origin/gh/ydwu4/339/head 2025-12-04T09:20:54.5812028Z * [new branch] gh/ydwu4/339/orig -> origin/gh/ydwu4/339/orig 2025-12-04T09:20:54.5815234Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-12-04T09:20:54.5815569Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-12-04T09:20:54.5815732Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-12-04T09:20:54.5815872Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-12-04T09:20:54.5819770Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-12-04T09:20:54.5820128Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-12-04T09:20:54.5820364Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-12-04T09:20:54.5820534Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-12-04T09:20:54.5820773Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-12-04T09:20:54.5821389Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-12-04T09:20:54.5825408Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-12-04T09:20:54.5825744Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-12-04T09:20:54.5826183Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-12-04T09:20:54.5826341Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-12-04T09:20:54.5826585Z * [new branch] gh/yushangdi/1/base -> origin/gh/yushangdi/1/base 2025-12-04T09:20:54.5826802Z * [new branch] gh/yushangdi/1/head -> origin/gh/yushangdi/1/head 2025-12-04T09:20:54.5828336Z * [new branch] gh/yushangdi/10/base -> origin/gh/yushangdi/10/base 2025-12-04T09:20:54.5828679Z * [new branch] gh/yushangdi/10/head -> origin/gh/yushangdi/10/head 2025-12-04T09:20:54.5829045Z * [new branch] gh/yushangdi/10/orig -> origin/gh/yushangdi/10/orig 2025-12-04T09:20:54.5830727Z * [new branch] gh/yushangdi/11/base -> origin/gh/yushangdi/11/base 2025-12-04T09:20:54.5830915Z * [new branch] gh/yushangdi/11/head -> origin/gh/yushangdi/11/head 2025-12-04T09:20:54.5833057Z * [new branch] gh/yushangdi/11/orig -> origin/gh/yushangdi/11/orig 2025-12-04T09:20:54.5839663Z * [new branch] gh/yushangdi/2/base -> origin/gh/yushangdi/2/base 2025-12-04T09:20:54.5845520Z * [new branch] gh/yushangdi/2/head -> origin/gh/yushangdi/2/head 2025-12-04T09:20:54.5847813Z * [new branch] gh/yushangdi/7/base -> origin/gh/yushangdi/7/base 2025-12-04T09:20:54.5848115Z * [new branch] gh/yushangdi/7/head -> origin/gh/yushangdi/7/head 2025-12-04T09:20:54.5848321Z * [new branch] gh/yushangdi/7/orig -> origin/gh/yushangdi/7/orig 2025-12-04T09:20:54.5848473Z * [new branch] gh/yushangdi/8/base -> origin/gh/yushangdi/8/base 2025-12-04T09:20:54.5848710Z * [new branch] gh/yushangdi/8/head -> origin/gh/yushangdi/8/head 2025-12-04T09:20:54.5848944Z * [new branch] gh/yushangdi/8/orig -> origin/gh/yushangdi/8/orig 2025-12-04T09:20:54.5849096Z * [new branch] gh/yushangdi/9/base -> origin/gh/yushangdi/9/base 2025-12-04T09:20:54.5849238Z * [new branch] gh/yushangdi/9/head -> origin/gh/yushangdi/9/head 2025-12-04T09:20:54.5849393Z * [new branch] gh/yushangdi/9/orig -> origin/gh/yushangdi/9/orig 2025-12-04T09:20:54.5849551Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-12-04T09:20:54.5849869Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-12-04T09:20:54.5850022Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-12-04T09:20:54.5850169Z * [new branch] gh/zklaus/20/base -> origin/gh/zklaus/20/base 2025-12-04T09:20:54.5850306Z * [new branch] gh/zklaus/20/head -> origin/gh/zklaus/20/head 2025-12-04T09:20:54.5850442Z * [new branch] gh/zklaus/20/orig -> origin/gh/zklaus/20/orig 2025-12-04T09:20:54.5850592Z * [new branch] gh/zklaus/21/base -> origin/gh/zklaus/21/base 2025-12-04T09:20:54.5850730Z * [new branch] gh/zklaus/21/head -> origin/gh/zklaus/21/head 2025-12-04T09:20:54.5850870Z * [new branch] gh/zklaus/21/orig -> origin/gh/zklaus/21/orig 2025-12-04T09:20:54.5851014Z * [new branch] gh/zklaus/22/base -> origin/gh/zklaus/22/base 2025-12-04T09:20:54.5851152Z * [new branch] gh/zklaus/22/head -> origin/gh/zklaus/22/head 2025-12-04T09:20:54.5851382Z * [new branch] gh/zklaus/22/orig -> origin/gh/zklaus/22/orig 2025-12-04T09:20:54.5858730Z * [new branch] gh/zklaus/23/base -> origin/gh/zklaus/23/base 2025-12-04T09:20:54.5860992Z * [new branch] gh/zklaus/23/head -> origin/gh/zklaus/23/head 2025-12-04T09:20:54.5865781Z * [new branch] gh/zklaus/23/orig -> origin/gh/zklaus/23/orig 2025-12-04T09:20:54.5870784Z * [new branch] gh/zklaus/24/base -> origin/gh/zklaus/24/base 2025-12-04T09:20:54.5875923Z * [new branch] gh/zklaus/24/head -> origin/gh/zklaus/24/head 2025-12-04T09:20:54.5881076Z * [new branch] gh/zklaus/24/orig -> origin/gh/zklaus/24/orig 2025-12-04T09:20:54.5885642Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-12-04T09:20:54.5887979Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-12-04T09:20:54.5888201Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-12-04T09:20:54.5888352Z * [new branch] gh/zou3519/1199/base -> origin/gh/zou3519/1199/base 2025-12-04T09:20:54.5888496Z * [new branch] gh/zou3519/1199/head -> origin/gh/zou3519/1199/head 2025-12-04T09:20:54.5888647Z * [new branch] gh/zou3519/1199/orig -> origin/gh/zou3519/1199/orig 2025-12-04T09:20:54.5888805Z * [new branch] gh/zou3519/1200/base -> origin/gh/zou3519/1200/base 2025-12-04T09:20:54.5888947Z * [new branch] gh/zou3519/1200/head -> origin/gh/zou3519/1200/head 2025-12-04T09:20:54.5889096Z * [new branch] gh/zou3519/1200/orig -> origin/gh/zou3519/1200/orig 2025-12-04T09:20:54.5889256Z * [new branch] gh/zou3519/1201/base -> origin/gh/zou3519/1201/base 2025-12-04T09:20:54.5889402Z * [new branch] gh/zou3519/1201/head -> origin/gh/zou3519/1201/head 2025-12-04T09:20:54.5889536Z * [new branch] gh/zou3519/1201/orig -> origin/gh/zou3519/1201/orig 2025-12-04T09:20:54.5889671Z * [new branch] gh/zou3519/1202/base -> origin/gh/zou3519/1202/base 2025-12-04T09:20:54.5889813Z * [new branch] gh/zou3519/1202/head -> origin/gh/zou3519/1202/head 2025-12-04T09:20:54.5889952Z * [new branch] gh/zou3519/1202/orig -> origin/gh/zou3519/1202/orig 2025-12-04T09:20:54.5890111Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-12-04T09:20:54.5890254Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-12-04T09:20:54.5890409Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-12-04T09:20:54.5890569Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-12-04T09:20:54.5890703Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-12-04T09:20:54.5890844Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-12-04T09:20:54.5890976Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-12-04T09:20:54.5891107Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-12-04T09:20:54.5891250Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-12-04T09:20:54.5891385Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-12-04T09:20:54.5891746Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-12-04T09:20:54.5891894Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-12-04T09:20:54.5892028Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-12-04T09:20:54.5892169Z * [new branch] gh/zpcore/14/orig -> origin/gh/zpcore/14/orig 2025-12-04T09:20:54.5892300Z * [new branch] gh/zpcore/15/base -> origin/gh/zpcore/15/base 2025-12-04T09:20:54.5892432Z * [new branch] gh/zpcore/15/head -> origin/gh/zpcore/15/head 2025-12-04T09:20:54.5892572Z * [new branch] gh/zpcore/15/orig -> origin/gh/zpcore/15/orig 2025-12-04T09:20:54.5892934Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-12-04T09:20:54.5897687Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-12-04T09:20:54.5903201Z * [new branch] gh/zpcore/21/base -> origin/gh/zpcore/21/base 2025-12-04T09:20:54.5908542Z * [new branch] gh/zpcore/21/head -> origin/gh/zpcore/21/head 2025-12-04T09:20:54.5912430Z * [new branch] gh/zpcore/21/orig -> origin/gh/zpcore/21/orig 2025-12-04T09:20:54.5916591Z * [new branch] gh/zpcore/22/base -> origin/gh/zpcore/22/base 2025-12-04T09:20:54.5921010Z * [new branch] gh/zpcore/22/head -> origin/gh/zpcore/22/head 2025-12-04T09:20:54.5925173Z * [new branch] gh/zpcore/22/orig -> origin/gh/zpcore/22/orig 2025-12-04T09:20:54.5928292Z * [new branch] gh/zpcore/23/base -> origin/gh/zpcore/23/base 2025-12-04T09:20:54.5928488Z * [new branch] gh/zpcore/23/head -> origin/gh/zpcore/23/head 2025-12-04T09:20:54.5928640Z * [new branch] gh/zpcore/23/orig -> origin/gh/zpcore/23/orig 2025-12-04T09:20:54.5928772Z * [new branch] gh/zpcore/24/base -> origin/gh/zpcore/24/base 2025-12-04T09:20:54.5928914Z * [new branch] gh/zpcore/24/head -> origin/gh/zpcore/24/head 2025-12-04T09:20:54.5929060Z * [new branch] gh/zpcore/24/orig -> origin/gh/zpcore/24/orig 2025-12-04T09:20:54.5929196Z * [new branch] gh/zpcore/25/base -> origin/gh/zpcore/25/base 2025-12-04T09:20:54.5929372Z * [new branch] gh/zpcore/25/head -> origin/gh/zpcore/25/head 2025-12-04T09:20:54.5929505Z * [new branch] gh/zpcore/25/orig -> origin/gh/zpcore/25/orig 2025-12-04T09:20:54.5929640Z * [new branch] gh/zpcore/26/base -> origin/gh/zpcore/26/base 2025-12-04T09:20:54.5929796Z * [new branch] gh/zpcore/26/head -> origin/gh/zpcore/26/head 2025-12-04T09:20:54.5929950Z * [new branch] gh/zpcore/26/orig -> origin/gh/zpcore/26/orig 2025-12-04T09:20:54.5930088Z * [new branch] gh/zpcore/27/base -> origin/gh/zpcore/27/base 2025-12-04T09:20:54.5930228Z * [new branch] gh/zpcore/27/head -> origin/gh/zpcore/27/head 2025-12-04T09:20:54.5930363Z * [new branch] gh/zpcore/27/orig -> origin/gh/zpcore/27/orig 2025-12-04T09:20:54.5930505Z * [new branch] gh/zpcore/28/base -> origin/gh/zpcore/28/base 2025-12-04T09:20:54.5930641Z * [new branch] gh/zpcore/28/head -> origin/gh/zpcore/28/head 2025-12-04T09:20:54.5930783Z * [new branch] gh/zpcore/28/orig -> origin/gh/zpcore/28/orig 2025-12-04T09:20:54.5930935Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-12-04T09:20:54.5931077Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-12-04T09:20:54.5931216Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-12-04T09:20:54.5931350Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-12-04T09:20:54.5931748Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-12-04T09:20:54.5931912Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-12-04T09:20:54.5932050Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-12-04T09:20:54.5932191Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-12-04T09:20:54.5932327Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-12-04T09:20:54.5932463Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-12-04T09:20:54.5932764Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-12-04T09:20:54.5932898Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-12-04T09:20:54.5933044Z * [new branch] google-main -> origin/google-main 2025-12-04T09:20:54.5933293Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-12-04T09:20:54.5933459Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-12-04T09:20:54.5933740Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-12-04T09:20:54.5933972Z * [new branch] hameerabbasi/complex_tensor_subclass -> origin/hameerabbasi/complex_tensor_subclass 2025-12-04T09:20:54.5934231Z * [new branch] hameerabbasi/fix-ctensor-gradcheck-tests -> origin/hameerabbasi/fix-ctensor-gradcheck-tests 2025-12-04T09:20:54.5934436Z * [new branch] hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose 2025-12-04T09:20:54.5934585Z * [new branch] hc_baseline -> origin/hc_baseline 2025-12-04T09:20:54.5934720Z * [new branch] hhh_rand -> origin/hhh_rand 2025-12-04T09:20:54.5934857Z * [new branch] huba/f1 -> origin/huba/f1 2025-12-04T09:20:54.5935248Z * [new branch] increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test -> origin/increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test 2025-12-04T09:20:54.5935393Z * [new branch] inlining -> origin/inlining 2025-12-04T09:20:54.5935540Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-12-04T09:20:54.5935716Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-12-04T09:20:54.5936038Z * [new branch] instrument-trunk-pull-linux-with-job-test-filters -> origin/instrument-trunk-pull-linux-with-job-test-filters 2025-12-04T09:20:54.5936191Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-12-04T09:20:54.5940832Z * [new branch] issue#58739 -> origin/issue#58739 2025-12-04T09:20:54.5941872Z * [new branch] jainapurva-patch-1 -> origin/jainapurva-patch-1 2025-12-04T09:20:54.5942133Z * [new branch] jathu/o3 -> origin/jathu/o3 2025-12-04T09:20:54.5942275Z * [new branch] jathu/sve -> origin/jathu/sve 2025-12-04T09:20:54.5942512Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-12-04T09:20:54.5942800Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-12-04T09:20:54.5943032Z * [new branch] jiannanWang/memorysnapshot_filter -> origin/jiannanWang/memorysnapshot_filter 2025-12-04T09:20:54.5943357Z * [new branch] jiannanWang/profilerstepwarning -> origin/jiannanWang/profilerstepwarning 2025-12-04T09:20:54.5943590Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-12-04T09:20:54.5943820Z * [new branch] jithunnair-amd-patch-10 -> origin/jithunnair-amd-patch-10 2025-12-04T09:20:54.5944051Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-12-04T09:20:54.5946001Z * [new branch] jithunnair-amd-patch-3 -> origin/jithunnair-amd-patch-3 2025-12-04T09:20:54.5946384Z * [new branch] jithunnair-amd-patch-4 -> origin/jithunnair-amd-patch-4 2025-12-04T09:20:54.5946692Z * [new branch] jithunnair-amd-patch-5 -> origin/jithunnair-amd-patch-5 2025-12-04T09:20:54.5946947Z * [new branch] jithunnair-amd-patch-6 -> origin/jithunnair-amd-patch-6 2025-12-04T09:20:54.5947238Z * [new branch] jithunnair-amd-patch-7 -> origin/jithunnair-amd-patch-7 2025-12-04T09:20:54.5947708Z * [new branch] jithunnair-amd-patch-8 -> origin/jithunnair-amd-patch-8 2025-12-04T09:20:54.5948393Z * [new branch] jithunnair-amd-patch-9 -> origin/jithunnair-amd-patch-9 2025-12-04T09:20:54.5949760Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-12-04T09:20:54.5950154Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-12-04T09:20:54.5950405Z * [new branch] kainan_test -> origin/kainan_test 2025-12-04T09:20:54.5950576Z * [new branch] larryliu0820-patch-1 -> origin/larryliu0820-patch-1 2025-12-04T09:20:54.5950865Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-12-04T09:20:54.5957571Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-12-04T09:20:54.5962067Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-12-04T09:20:54.5964192Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-12-04T09:20:54.5964527Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-12-04T09:20:54.5964810Z * [new branch] llama4-stable -> origin/llama4-stable 2025-12-04T09:20:54.5964964Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-12-04T09:20:54.5965194Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-12-04T09:20:54.5965337Z * [new branch] lucaskabela/fix_164876 -> origin/lucaskabela/fix_164876 2025-12-04T09:20:54.5965503Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-12-04T09:20:54.5965672Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-12-04T09:20:54.5965864Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-12-04T09:20:54.5966086Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-12-04T09:20:54.5966294Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-12-04T09:20:54.5966522Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-12-04T09:20:54.5966667Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-12-04T09:20:54.5966825Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-12-04T09:20:54.5967000Z * [new branch] lucaskabela/typing_ctx_manager -> origin/lucaskabela/typing_ctx_manager 2025-12-04T09:20:54.5967164Z * [new branch] lucaskabela/typing_nn_module -> origin/lucaskabela/typing_nn_module 2025-12-04T09:20:54.5967343Z * [new branch] lucaskabela/typing_user_defined -> origin/lucaskabela/typing_user_defined 2025-12-04T09:20:54.5967501Z * [new branch] lucaskabela/typing_variables -> origin/lucaskabela/typing_variables 2025-12-04T09:20:54.5967687Z * [new branch] lucaskabela/typing_variables_dicts -> origin/lucaskabela/typing_variables_dicts 2025-12-04T09:20:54.5967898Z * [new branch] lucaskabela/typing_variables_functions -> origin/lucaskabela/typing_variables_functions 2025-12-04T09:20:54.5968080Z * [new branch] lucaskabela/typing_variables_lists -> origin/lucaskabela/typing_variables_lists 2025-12-04T09:20:54.5968220Z * [new branch] lw/torch_box_by_ref -> origin/lw/torch_box_by_ref 2025-12-04T09:20:54.5968779Z * [new branch] main -> origin/main 2025-12-04T09:20:54.5969119Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-12-04T09:20:54.5969386Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-12-04T09:20:54.5969528Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-12-04T09:20:54.5970573Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-12-04T09:20:54.5970997Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-12-04T09:20:54.5972200Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-12-04T09:20:54.5972755Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-12-04T09:20:54.5974056Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-12-04T09:20:54.5974405Z * [new branch] malfet/add-3.14-ci -> origin/malfet/add-3.14-ci 2025-12-04T09:20:54.5975879Z * [new branch] malfet/be-do-not-make-typos-in-build-artifacts -> origin/malfet/be-do-not-make-typos-in-build-artifacts 2025-12-04T09:20:54.5976213Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-12-04T09:20:54.5978419Z * [new branch] malfet/be-remove-misisng-neon-headers -> origin/malfet/be-remove-misisng-neon-headers 2025-12-04T09:20:54.5982494Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-12-04T09:20:54.5986581Z * [new branch] manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe 2025-12-04T09:20:54.5991358Z * [new branch] manuel/inductor_link_openmp -> origin/manuel/inductor_link_openmp 2025-12-04T09:20:54.5996373Z * [new branch] masnesral/metaconda -> origin/masnesral/metaconda 2025-12-04T09:20:54.5998393Z * [new branch] mem_profiler_flaky_fix -> origin/mem_profiler_flaky_fix 2025-12-04T09:20:54.5998600Z * [new branch] mem_profiler_stack_trace -> origin/mem_profiler_stack_trace 2025-12-04T09:20:54.5998752Z * [new branch] memory_profiler_stack -> origin/memory_profiler_stack 2025-12-04T09:20:54.5998920Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-12-04T09:20:54.5999061Z * [new branch] mingw_posix -> origin/mingw_posix 2025-12-04T09:20:54.5999212Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-12-04T09:20:54.5999340Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-12-04T09:20:54.5999466Z * [new branch] mlazos/acts -> origin/mlazos/acts 2025-12-04T09:20:54.5999612Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-12-04T09:20:54.5999781Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-12-04T09:20:54.5999980Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-12-04T09:20:54.6000132Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-12-04T09:20:54.6000266Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-12-04T09:20:54.6000401Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-12-04T09:20:54.6000538Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-12-04T09:20:54.6000659Z * [new branch] mlazos/bwd -> origin/mlazos/bwd 2025-12-04T09:20:54.6000812Z * [new branch] mlazos/combo-test -> origin/mlazos/combo-test 2025-12-04T09:20:54.6002486Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-12-04T09:20:54.6002686Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-12-04T09:20:54.6003142Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-12-04T09:20:54.6008526Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-12-04T09:20:54.6008730Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-12-04T09:20:54.6009120Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-12-04T09:20:54.6009305Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-12-04T09:20:54.6009454Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-12-04T09:20:54.6009592Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-12-04T09:20:54.6009739Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-12-04T09:20:54.6009882Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-12-04T09:20:54.6010036Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-12-04T09:20:54.6010181Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-12-04T09:20:54.6010305Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-12-04T09:20:54.6010478Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-12-04T09:20:54.6010619Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-12-04T09:20:54.6010742Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-12-04T09:20:54.6010881Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-12-04T09:20:54.6011037Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-12-04T09:20:54.6011173Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-12-04T09:20:54.6011326Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-12-04T09:20:54.6011680Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-12-04T09:20:54.6011839Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-12-04T09:20:54.6011981Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-12-04T09:20:54.6012410Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-12-04T09:20:54.6016552Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-12-04T09:20:54.6016887Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-12-04T09:20:54.6017058Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-12-04T09:20:54.6017190Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-12-04T09:20:54.6017483Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-12-04T09:20:54.6017955Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-12-04T09:20:54.6018114Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-12-04T09:20:54.6018270Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-12-04T09:20:54.6019150Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-12-04T09:20:54.6019359Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-12-04T09:20:54.6021921Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-12-04T09:20:54.6022079Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-12-04T09:20:54.6028602Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-12-04T09:20:54.6032751Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-12-04T09:20:54.6036840Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-12-04T09:20:54.6041147Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-12-04T09:20:54.6045028Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-12-04T09:20:54.6047771Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-12-04T09:20:54.6047923Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-12-04T09:20:54.6048067Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-12-04T09:20:54.6048234Z * [new branch] mlazos/inductor-streams -> origin/mlazos/inductor-streams 2025-12-04T09:20:54.6048356Z * [new branch] mlazos/main -> origin/mlazos/main 2025-12-04T09:20:54.6048467Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-12-04T09:20:54.6048615Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-12-04T09:20:54.6048798Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-12-04T09:20:54.6048973Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-12-04T09:20:54.6049097Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-12-04T09:20:54.6049220Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-12-04T09:20:54.6049347Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-12-04T09:20:54.6049483Z * [new branch] mlazos/overguarding -> origin/mlazos/overguarding 2025-12-04T09:20:54.6049626Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-12-04T09:20:54.6049754Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-12-04T09:20:54.6049881Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-12-04T09:20:54.6050017Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-12-04T09:20:54.6050136Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-12-04T09:20:54.6050250Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-12-04T09:20:54.6050366Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-12-04T09:20:54.6050506Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-12-04T09:20:54.6050661Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-12-04T09:20:54.6050777Z * [new branch] mlazos/stests -> origin/mlazos/stests 2025-12-04T09:20:54.6050907Z * [new branch] mlazos/stream-ops -> origin/mlazos/stream-ops 2025-12-04T09:20:54.6051025Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-12-04T09:20:54.6051165Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-12-04T09:20:54.6051279Z * [new branch] mlazos/test -> origin/mlazos/test 2025-12-04T09:20:54.6051397Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-12-04T09:20:54.6051792Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-12-04T09:20:54.6051954Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-12-04T09:20:54.6052090Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-12-04T09:20:54.6052228Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-12-04T09:20:54.6052436Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-12-04T09:20:54.6052577Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-12-04T09:20:54.6052728Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-12-04T09:20:54.6052913Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-12-04T09:20:54.6053085Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-12-04T09:20:54.6053241Z * [new branch] mlazos/user-stream-base -> origin/mlazos/user-stream-base 2025-12-04T09:20:54.6053380Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-12-04T09:20:54.6053561Z * [new branch] mlazos/user-streams-backup -> origin/mlazos/user-streams-backup 2025-12-04T09:20:54.6053730Z * [new branch] mlazos/user-streams-backup2 -> origin/mlazos/user-streams-backup2 2025-12-04T09:20:54.6054270Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-12-04T09:20:54.6054624Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-12-04T09:20:54.6055188Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-12-04T09:20:54.6056295Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-12-04T09:20:54.6056925Z * [new branch] module-shim -> origin/module-shim 2025-12-04T09:20:54.6058200Z * [new branch] move_config -> origin/move_config 2025-12-04T09:20:54.6058713Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-12-04T09:20:54.6062014Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-12-04T09:20:54.6062240Z * [new branch] mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape 2025-12-04T09:20:54.6062393Z * [new branch] my_varlen_backup -> origin/my_varlen_backup 2025-12-04T09:20:54.6062540Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-12-04T09:20:54.6062977Z * [new branch] new-codegen -> origin/new-codegen 2025-12-04T09:20:54.6063899Z * [new branch] newtest-base -> origin/newtest-base 2025-12-04T09:20:54.6065176Z * [new branch] ngimel/addmm_dtype -> origin/ngimel/addmm_dtype 2025-12-04T09:20:54.6065314Z * [new branch] ngimel/div_inv -> origin/ngimel/div_inv 2025-12-04T09:20:54.6065885Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-12-04T09:20:54.6066755Z * [new branch] ngimel/gather_grid -> origin/ngimel/gather_grid 2025-12-04T09:20:54.6067080Z * [new branch] ngimel/gather_grid_release -> origin/ngimel/gather_grid_release 2025-12-04T09:20:54.6068296Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-12-04T09:20:54.6068444Z * [new branch] ngimel/hostalloc -> origin/ngimel/hostalloc 2025-12-04T09:20:54.6070544Z * [new branch] ngimel/storage_id -> origin/ngimel/storage_id 2025-12-04T09:20:54.6070849Z * [new branch] nightly -> origin/nightly 2025-12-04T09:20:54.6071161Z * [new branch] nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check 2025-12-04T09:20:54.6071749Z * [new branch] nikitaved/addmm_epilogue_fusions_2d_bias -> origin/nikitaved/addmm_epilogue_fusions_2d_bias 2025-12-04T09:20:54.6072705Z * [new branch] nikitaved/addmm_epilogue_fusions_inductor -> origin/nikitaved/addmm_epilogue_fusions_inductor 2025-12-04T09:20:54.6078500Z * [new branch] nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch 2025-12-04T09:20:54.6080967Z * [new branch] nikitaved/grad_addmm_epilogue_fusions -> origin/nikitaved/grad_addmm_epilogue_fusions 2025-12-04T09:20:54.6081177Z * [new branch] nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index 2025-12-04T09:20:54.6081427Z * [new branch] nikitaved/test -> origin/nikitaved/test 2025-12-04T09:20:54.6081668Z * [new branch] nmacchioni-perf-test-async-autotune -> origin/nmacchioni-perf-test-async-autotune 2025-12-04T09:20:54.6081817Z * [new branch] no_distributed_log_spew -> origin/no_distributed_log_spew 2025-12-04T09:20:54.6081948Z * [new branch] nofun-hack -> origin/nofun-hack 2025-12-04T09:20:54.6082062Z * [new branch] norm_bench -> origin/norm_bench 2025-12-04T09:20:54.6082205Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-12-04T09:20:54.6082352Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-12-04T09:20:54.6082478Z * [new branch] optimizer_test -> origin/optimizer_test 2025-12-04T09:20:54.6084802Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-12-04T09:20:54.6085013Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-12-04T09:20:54.6085164Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-12-04T09:20:54.6085326Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-12-04T09:20:54.6085463Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-12-04T09:20:54.6086458Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-12-04T09:20:54.6086747Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-12-04T09:20:54.6087888Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-12-04T09:20:54.6088220Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-12-04T09:20:54.6089226Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-12-04T09:20:54.6089630Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-12-04T09:20:54.6090650Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-12-04T09:20:54.6091606Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-12-04T09:20:54.6091925Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-12-04T09:20:54.6092856Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-12-04T09:20:54.6094300Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-12-04T09:20:54.6095206Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-12-04T09:20:54.6095457Z * [new branch] orig/release/2.9 -> origin/orig/release/2.9 2025-12-04T09:20:54.6097824Z * [new branch] origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base 2025-12-04T09:20:54.6098225Z * [new branch] origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig 2025-12-04T09:20:54.6099643Z * [new branch] origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig 2025-12-04T09:20:54.6100645Z * [new branch] oulgen-patch-1 -> origin/oulgen-patch-1 2025-12-04T09:20:54.6101220Z * [new branch] oulgen-patch-2 -> origin/oulgen-patch-2 2025-12-04T09:20:54.6102246Z * [new branch] oulgen-patch-3 -> origin/oulgen-patch-3 2025-12-04T09:20:54.6102697Z * [new branch] oulgen-patch-4 -> origin/oulgen-patch-4 2025-12-04T09:20:54.6104016Z * [new branch] padded-tensor -> origin/padded-tensor 2025-12-04T09:20:54.6105115Z * [new branch] pca2 -> origin/pca2 2025-12-04T09:20:54.6108018Z * [new branch] per_channel_backup -> origin/per_channel_backup 2025-12-04T09:20:54.6108725Z * [new branch] perf_ops -> origin/perf_ops 2025-12-04T09:20:54.6108967Z * [new branch] perf_ops_2_9 -> origin/perf_ops_2_9 2025-12-04T09:20:54.6109151Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-12-04T09:20:54.6113382Z * [new branch] pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode 2025-12-04T09:20:54.6114021Z * [new branch] pianpwk/_debug_mode_for_triton_draft -> origin/pianpwk/_debug_mode_for_triton_draft 2025-12-04T09:20:54.6114314Z * [new branch] pianpwk/_debug_nn_module_compile -> origin/pianpwk/_debug_nn_module_compile 2025-12-04T09:20:54.6114660Z * [new branch] pianpwk/_draft_triton_11_3 -> origin/pianpwk/_draft_triton_11_3 2025-12-04T09:20:54.6114847Z * [new branch] pianpwk/_manual_bucket_draft -> origin/pianpwk/_manual_bucket_draft 2025-12-04T09:20:54.6115068Z * [new branch] pianpwk/_profile_w_dispatch_keys -> origin/pianpwk/_profile_w_dispatch_keys 2025-12-04T09:20:54.6115243Z * [new branch] pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode 2025-12-04T09:20:54.6115446Z * [new branch] pianpwk/_unbacked_local_shard_size -> origin/pianpwk/_unbacked_local_shard_size 2025-12-04T09:20:54.6117149Z * [new branch] pianpwk/anomaly_tb -> origin/pianpwk/anomaly_tb 2025-12-04T09:20:54.6117421Z * [new branch] pianpwk/auto_fx_annotate -> origin/pianpwk/auto_fx_annotate 2025-12-04T09:20:54.6117631Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-12-04T09:20:54.6117807Z * [new branch] pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf 2025-12-04T09:20:54.6117983Z * [new branch] pianpwk/debug_fwd_stack_traces -> origin/pianpwk/debug_fwd_stack_traces 2025-12-04T09:20:54.6118144Z * [new branch] pianpwk/debug_hash_tensor -> origin/pianpwk/debug_hash_tensor 2025-12-04T09:20:54.6124239Z * [new branch] pianpwk/debug_mode_annotate -> origin/pianpwk/debug_mode_annotate 2025-12-04T09:20:54.6128485Z * [new branch] pianpwk/debug_mode_defaults -> origin/pianpwk/debug_mode_defaults 2025-12-04T09:20:54.6128689Z * [new branch] pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks 2025-12-04T09:20:54.6128896Z * [new branch] pianpwk/debug_mode_opcall_refactor -> origin/pianpwk/debug_mode_opcall_refactor 2025-12-04T09:20:54.6129078Z * [new branch] pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids 2025-12-04T09:20:54.6129272Z * [new branch] pianpwk/debug_mode_triton -> origin/pianpwk/debug_mode_triton 2025-12-04T09:20:54.6129451Z * [new branch] pianpwk/debug_show_stack_trace -> origin/pianpwk/debug_show_stack_trace 2025-12-04T09:20:54.6129640Z * [new branch] pianpwk/debug_wait_on_collective -> origin/pianpwk/debug_wait_on_collective 2025-12-04T09:20:54.6129853Z * [new branch] pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf 2025-12-04T09:20:54.6130089Z * [new branch] pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug 2025-12-04T09:20:54.6130289Z * [new branch] pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile 2025-12-04T09:20:54.6130474Z * [new branch] pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn 2025-12-04T09:20:54.6130689Z * [new branch] pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5 2025-12-04T09:20:54.6131019Z * [new branch] pianpwk/dtensor_custom_chunk -> origin/pianpwk/dtensor_custom_chunk 2025-12-04T09:20:54.6131219Z * [new branch] pianpwk/dtensor_unbacked_keypath -> origin/pianpwk/dtensor_unbacked_keypath 2025-12-04T09:20:54.6131425Z * [new branch] pianpwk/event_list_tree -> origin/pianpwk/event_list_tree 2025-12-04T09:20:54.6131777Z * [new branch] pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs 2025-12-04T09:20:54.6131941Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-12-04T09:20:54.6132151Z * [new branch] pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft 2025-12-04T09:20:54.6132371Z * [new branch] pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat 2025-12-04T09:20:54.6132588Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-12-04T09:20:54.6132769Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-12-04T09:20:54.6137906Z * [new branch] pianpwk/skip_python_keys_alternate -> origin/pianpwk/skip_python_keys_alternate 2025-12-04T09:20:54.6142758Z * [new branch] pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards 2025-12-04T09:20:54.6148418Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-12-04T09:20:54.6153714Z * [new branch] pianpwk/symint_one_hot -> origin/pianpwk/symint_one_hot 2025-12-04T09:20:54.6159816Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-12-04T09:20:54.6162511Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-12-04T09:20:54.6162793Z * [new branch] pianpwk/try_dumb_stuff -> origin/pianpwk/try_dumb_stuff 2025-12-04T09:20:54.6166127Z * [new branch] pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2 2025-12-04T09:20:54.6166349Z * [new branch] pianpwk/unbacked_dtensor_mm -> origin/pianpwk/unbacked_dtensor_mm 2025-12-04T09:20:54.6166533Z * [new branch] pianpwk/unbacked_tracing_12_2 -> origin/pianpwk/unbacked_tracing_12_2 2025-12-04T09:20:54.6166684Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-12-04T09:20:54.6166828Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-12-04T09:20:54.6166986Z * [new branch] piz/fix_partial_backward_1112 -> origin/piz/fix_partial_backward_1112 2025-12-04T09:20:54.6167130Z * [new branch] piz/prop_cache_clean -> origin/piz/prop_cache_clean 2025-12-04T09:20:54.6167257Z * [new branch] pool-separate -> origin/pool-separate 2025-12-04T09:20:54.6167395Z * [new branch] pr-156087 -> origin/pr-156087 2025-12-04T09:20:54.6167501Z * [new branch] pr/131860 -> origin/pr/131860 2025-12-04T09:20:54.6167627Z * [new branch] predispatch_to -> origin/predispatch_to 2025-12-04T09:20:54.6167754Z * [new branch] protect-c17 -> origin/protect-c17 2025-12-04T09:20:54.6167881Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-12-04T09:20:54.6168029Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-12-04T09:20:54.6168238Z * [new branch] q1l1/fix_device_moved_constant_type_unknown -> origin/q1l1/fix_device_moved_constant_type_unknown 2025-12-04T09:20:54.6168459Z * [new branch] q1l1/fix_wrong_default_type_for_kernel_call_args -> origin/q1l1/fix_wrong_default_type_for_kernel_call_args 2025-12-04T09:20:54.6168757Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-12-04T09:20:54.6168889Z * [new branch] quote-pytest_cache -> origin/quote-pytest_cache 2025-12-04T09:20:54.6169053Z * [new branch] reland-accgrad-stream-warn -> origin/reland-accgrad-stream-warn 2025-12-04T09:20:54.6169219Z * [new branch] release/1.10 -> origin/release/1.10 2025-12-04T09:20:54.6169331Z * [new branch] release/1.11 -> origin/release/1.11 2025-12-04T09:20:54.6169442Z * [new branch] release/1.12 -> origin/release/1.12 2025-12-04T09:20:54.6169551Z * [new branch] release/1.13 -> origin/release/1.13 2025-12-04T09:20:54.6169664Z * [new branch] release/1.4 -> origin/release/1.4 2025-12-04T09:20:54.6169784Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-12-04T09:20:54.6169894Z * [new branch] release/1.5 -> origin/release/1.5 2025-12-04T09:20:54.6170005Z * [new branch] release/1.6 -> origin/release/1.6 2025-12-04T09:20:54.6170108Z * [new branch] release/1.7 -> origin/release/1.7 2025-12-04T09:20:54.6170223Z * [new branch] release/1.8 -> origin/release/1.8 2025-12-04T09:20:54.6170339Z * [new branch] release/1.9 -> origin/release/1.9 2025-12-04T09:20:54.6170446Z * [new branch] release/2.0 -> origin/release/2.0 2025-12-04T09:20:54.6170560Z * [new branch] release/2.1 -> origin/release/2.1 2025-12-04T09:20:54.6170663Z * [new branch] release/2.2 -> origin/release/2.2 2025-12-04T09:20:54.6170768Z * [new branch] release/2.3 -> origin/release/2.3 2025-12-04T09:20:54.6170881Z * [new branch] release/2.4 -> origin/release/2.4 2025-12-04T09:20:54.6170992Z * [new branch] release/2.5 -> origin/release/2.5 2025-12-04T09:20:54.6171094Z * [new branch] release/2.6 -> origin/release/2.6 2025-12-04T09:20:54.6171208Z * [new branch] release/2.7 -> origin/release/2.7 2025-12-04T09:20:54.6171325Z * [new branch] release/2.8 -> origin/release/2.8 2025-12-04T09:20:54.6171597Z * [new branch] release/2.9 -> origin/release/2.9 2025-12-04T09:20:54.6171737Z * [new branch] release_notes -> origin/release_notes 2025-12-04T09:20:54.6171888Z * [new branch] remove_pyinterpreter -> origin/remove_pyinterpreter 2025-12-04T09:20:54.6172125Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-12-04T09:20:54.6172537Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-12-04T09:20:54.6172772Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-12-04T09:20:54.6176958Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-12-04T09:20:54.6177333Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-12-04T09:20:54.6177544Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-12-04T09:20:54.6177828Z * [new branch] revert-152361-gh/fadara01/1/head -> origin/revert-152361-gh/fadara01/1/head 2025-12-04T09:20:54.6182722Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-12-04T09:20:54.6187462Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-12-04T09:20:54.6189517Z * [new branch] revert-hoo-invoke-subgraph -> origin/revert-hoo-invoke-subgraph 2025-12-04T09:20:54.6189775Z * [new branch] revert_always_build_distributed -> origin/revert_always_build_distributed 2025-12-04T09:20:54.6189907Z * [new branch] rms_norm_patch -> origin/rms_norm_patch 2025-12-04T09:20:54.6190296Z * [new branch] ruisi/fix_all_to_all_estimation -> origin/ruisi/fix_all_to_all_estimation 2025-12-04T09:20:54.6190451Z * [new branch] ruisi/fix_comm_estimation -> origin/ruisi/fix_comm_estimation 2025-12-04T09:20:54.6190631Z * [new branch] ruisi/fix_dynamic_shape_estimation -> origin/ruisi/fix_dynamic_shape_estimation 2025-12-04T09:20:54.6190801Z * [new branch] ruisi/fix_llama3_autobucketing -> origin/ruisi/fix_llama3_autobucketing 2025-12-04T09:20:54.6190971Z * [new branch] ruisi/fix_manual_bucketing_ep_pass -> origin/ruisi/fix_manual_bucketing_ep_pass 2025-12-04T09:20:54.6191123Z * [new branch] ruisi/manual_bucket_pass -> origin/ruisi/manual_bucket_pass 2025-12-04T09:20:54.6191370Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-12-04T09:20:54.6191524Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-12-04T09:20:54.6191674Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-12-04T09:20:54.6191787Z * [new branch] rzou/njt -> origin/rzou/njt 2025-12-04T09:20:54.6193924Z * [new branch] rzou/pca -> origin/rzou/pca 2025-12-04T09:20:54.6196447Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-12-04T09:20:54.6196709Z * [new branch] samplevllm -> origin/samplevllm 2025-12-04T09:20:54.6197462Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-12-04T09:20:54.6197770Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-12-04T09:20:54.6202896Z * [new branch] sapling-pr-archive-tushar00jain -> origin/sapling-pr-archive-tushar00jain 2025-12-04T09:20:54.6207652Z * [new branch] save -> origin/save 2025-12-04T09:20:54.6207951Z * [new branch] scaled_mm -> origin/scaled_mm 2025-12-04T09:20:54.6208106Z * [new branch] scan_attempt -> origin/scan_attempt 2025-12-04T09:20:54.6208314Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-12-04T09:20:54.6208596Z * [new branch] sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix 2025-12-04T09:20:54.6208880Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-12-04T09:20:54.6209571Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-12-04T09:20:54.6209764Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-12-04T09:20:54.6209936Z * [new branch] some_rocm_inductor_skips -> origin/some_rocm_inductor_skips 2025-12-04T09:20:54.6210147Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-12-04T09:20:54.6210325Z * [new branch] sparse-mm-bf16-support -> origin/sparse-mm-bf16-support 2025-12-04T09:20:54.6210478Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-12-04T09:20:54.6210608Z * [new branch] suo -> origin/suo 2025-12-04T09:20:54.6210735Z * [new branch] sve-poc -> origin/sve-poc 2025-12-04T09:20:54.6210869Z * [new branch] switch-bn -> origin/switch-bn 2025-12-04T09:20:54.6211218Z * [new branch] sy_annotation_in_autograd_hop -> origin/sy_annotation_in_autograd_hop 2025-12-04T09:20:54.6211366Z * [new branch] sy_aot_eager_record -> origin/sy_aot_eager_record 2025-12-04T09:20:54.6211699Z * [new branch] sy_custom_bucketing -> origin/sy_custom_bucketing 2025-12-04T09:20:54.6211933Z * [new branch] sy_debug_mode_test -> origin/sy_debug_mode_test 2025-12-04T09:20:54.6212085Z * [new branch] sy_deserialize -> origin/sy_deserialize 2025-12-04T09:20:54.6212213Z * [new branch] sy_dump_gm_code -> origin/sy_dump_gm_code 2025-12-04T09:20:54.6212328Z * [new branch] sy_exp -> origin/sy_exp 2025-12-04T09:20:54.6212485Z * [new branch] sy_export_annotation -> origin/sy_export_annotation 2025-12-04T09:20:54.6213260Z * [new branch] sy_invoke_subgraph -> origin/sy_invoke_subgraph 2025-12-04T09:20:54.6214135Z * [new branch] sy_kernel_bw_name -> origin/sy_kernel_bw_name 2025-12-04T09:20:54.6214527Z * [new branch] sy_multi_arch -> origin/sy_multi_arch 2025-12-04T09:20:54.6219134Z * [new branch] sy_nn_module_stack -> origin/sy_nn_module_stack 2025-12-04T09:20:54.6219488Z * [new branch] sy_original_dtensor -> origin/sy_original_dtensor 2025-12-04T09:20:54.6219712Z * [new branch] sy_profiler_cia -> origin/sy_profiler_cia 2025-12-04T09:20:54.6219875Z * [new branch] symm_mem_sync -> origin/symm_mem_sync 2025-12-04T09:20:54.6220042Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-12-04T09:20:54.6220191Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-12-04T09:20:54.6220480Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-12-04T09:20:54.6220622Z * [new branch] test-old -> origin/test-old 2025-12-04T09:20:54.6225874Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-12-04T09:20:54.6226194Z * [new branch] tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix 2025-12-04T09:20:54.6229955Z * [new branch] tianren/customOp_enable_max_autotune -> origin/tianren/customOp_enable_max_autotune 2025-12-04T09:20:54.6230286Z * [new branch] tianren/customOp_fusion -> origin/tianren/customOp_fusion 2025-12-04T09:20:54.6235922Z * [new branch] tianren/customop_collectiveop_benchmark -> origin/tianren/customop_collectiveop_benchmark 2025-12-04T09:20:54.6240072Z * [new branch] tianren/customop_collectiveop_benchmark_fix -> origin/tianren/customop_collectiveop_benchmark_fix 2025-12-04T09:20:54.6242053Z * [new branch] tianren/customop_dynamic_config -> origin/tianren/customop_dynamic_config 2025-12-04T09:20:54.6242257Z * [new branch] tianren/dynamic_range_input -> origin/tianren/dynamic_range_input 2025-12-04T09:20:54.6242435Z * [new branch] tianren/dynamic_range_input_fix -> origin/tianren/dynamic_range_input_fix 2025-12-04T09:20:54.6242639Z * [new branch] tianren/dynamic_range_input_merge -> origin/tianren/dynamic_range_input_merge 2025-12-04T09:20:54.6242805Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-12-04T09:20:54.6242959Z * [new branch] tianren/fx_codegen_dump -> origin/tianren/fx_codegen_dump 2025-12-04T09:20:54.6243113Z * [new branch] tianren/symmetric_memory -> origin/tianren/symmetric_memory 2025-12-04T09:20:54.6243233Z * [new branch] tianren/test -> origin/tianren/test 2025-12-04T09:20:54.6243381Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-12-04T09:20:54.6243628Z * [new branch] tmp -> origin/tmp 2025-12-04T09:20:54.6243759Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-12-04T09:20:54.6243912Z * [new branch] torchtitan_integration -> origin/torchtitan_integration 2025-12-04T09:20:54.6244100Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-12-04T09:20:54.6244249Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-12-04T09:20:54.6244377Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-12-04T09:20:54.6244499Z * [new branch] triton_kernel -> origin/triton_kernel 2025-12-04T09:20:54.6244611Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-12-04T09:20:54.6244725Z * [new branch] type_dec -> origin/type_dec 2025-12-04T09:20:54.6244898Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-12-04T09:20:54.6245126Z * [new branch] update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1 2025-12-04T09:20:54.6245341Z * [new branch] update-audio-commit-hash/19087141161-1916-1 -> origin/update-audio-commit-hash/19087141161-1916-1 2025-12-04T09:20:54.6245552Z * [new branch] update-audio-commit-hash/19250643381-1929-1 -> origin/update-audio-commit-hash/19250643381-1929-1 2025-12-04T09:20:54.6245758Z * [new branch] update-audio-commit-hash/19397724337-1935-1 -> origin/update-audio-commit-hash/19397724337-1935-1 2025-12-04T09:20:54.6245966Z * [new branch] update-audio-commit-hash/19555670148-1941-1 -> origin/update-audio-commit-hash/19555670148-1941-1 2025-12-04T09:20:54.6246167Z * [new branch] update-audio-commit-hash/19750627930-1946-1 -> origin/update-audio-commit-hash/19750627930-1946-1 2025-12-04T09:20:54.6247575Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-12-04T09:20:54.6247794Z * [new branch] update-vision-commit-hash/19087141161-1916-1 -> origin/update-vision-commit-hash/19087141161-1916-1 2025-12-04T09:20:54.6248509Z * [new branch] update-vision-commit-hash/19184897099-1925-1 -> origin/update-vision-commit-hash/19184897099-1925-1 2025-12-04T09:20:54.6248773Z * [new branch] update-vision-commit-hash/19250643381-1929-1 -> origin/update-vision-commit-hash/19250643381-1929-1 2025-12-04T09:20:54.6249010Z * [new branch] update-vision-commit-hash/19381328640-1934-1 -> origin/update-vision-commit-hash/19381328640-1934-1 2025-12-04T09:20:54.6249232Z * [new branch] update-vision-commit-hash/19485237164-1938-1 -> origin/update-vision-commit-hash/19485237164-1938-1 2025-12-04T09:20:54.6249476Z * [new branch] update-vllm-commit-hash/18451675449-1879-1 -> origin/update-vllm-commit-hash/18451675449-1879-1 2025-12-04T09:20:54.6249655Z * [new branch] update-vllm-dockerfile -> origin/update-vllm-dockerfile 2025-12-04T09:20:54.6249870Z * [new branch] update-xla-commit-hash/19224287370-211-1 -> origin/update-xla-commit-hash/19224287370-211-1 2025-12-04T09:20:54.6250084Z * [new branch] update-xla-commit-hash/19422028566-212-1 -> origin/update-xla-commit-hash/19422028566-212-1 2025-12-04T09:20:54.6250314Z * [new branch] update-xla-commit-hash/19626841311-213-1 -> origin/update-xla-commit-hash/19626841311-213-1 2025-12-04T09:20:54.6253619Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-12-04T09:20:54.6259803Z * [new branch] update_operator_readme -> origin/update_operator_readme 2025-12-04T09:20:54.6261914Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-12-04T09:20:54.6262155Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-12-04T09:20:54.6262398Z * [new branch] update_slow_tests_1762155677 -> origin/update_slow_tests_1762155677 2025-12-04T09:20:54.6262732Z * [new branch] update_slow_tests_1763365283 -> origin/update_slow_tests_1763365283 2025-12-04T09:20:54.6263005Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-12-04T09:20:54.6268634Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-12-04T09:20:54.6272924Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-12-04T09:20:54.6277173Z * [new branch] upload-tests-for-autorevert -> origin/upload-tests-for-autorevert 2025-12-04T09:20:54.6281280Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-12-04T09:20:54.6285411Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-12-04T09:20:54.6290622Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-12-04T09:20:54.6290792Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-12-04T09:20:54.6290929Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-12-04T09:20:54.6291036Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-12-04T09:20:54.6291148Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-12-04T09:20:54.6291274Z * [new branch] validate_fn -> origin/validate_fn 2025-12-04T09:20:54.6291419Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-12-04T09:20:54.6291803Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-12-04T09:20:54.6291974Z * [new branch] varlen-api -> origin/varlen-api 2025-12-04T09:20:54.6292140Z * [new branch] varlen-api-backup -> origin/varlen-api-backup 2025-12-04T09:20:54.6292302Z * [new branch] varlen_batch_invariance -> origin/varlen_batch_invariance 2025-12-04T09:20:54.6292436Z * [new branch] viable/strict -> origin/viable/strict 2025-12-04T09:20:54.6292667Z * [new branch] vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy 2025-12-04T09:20:54.6292791Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-12-04T09:20:54.6292899Z * [new branch] vllmpin -> origin/vllmpin 2025-12-04T09:20:54.6293073Z * [new branch] vscode-recommend-pyrefly -> origin/vscode-recommend-pyrefly 2025-12-04T09:20:54.6293203Z * [new branch] wdvr-patch-1 -> origin/wdvr-patch-1 2025-12-04T09:20:54.6293340Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-12-04T09:20:54.6293455Z * [new branch] whc/pei -> origin/whc/pei 2025-12-04T09:20:54.6293579Z * [new branch] whc/pp_fix -> origin/whc/pp_fix 2025-12-04T09:20:54.6293708Z * [new branch] whc/sharding -> origin/whc/sharding 2025-12-04T09:20:54.6293839Z * [new branch] whc/sharding2 -> origin/whc/sharding2 2025-12-04T09:20:54.6293961Z * [new branch] whc/uneven -> origin/whc/uneven 2025-12-04T09:20:54.6294097Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-12-04T09:20:54.6294216Z * [new branch] win_warnings -> origin/win_warnings 2025-12-04T09:20:54.6294370Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-12-04T09:20:54.6294488Z * [new branch] xmfan-war -> origin/xmfan-war 2025-12-04T09:20:54.6294829Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-12-04T09:20:54.6294974Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-12-04T09:20:54.6295293Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-12-04T09:20:54.6295439Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-12-04T09:20:54.6295574Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-12-04T09:20:54.6295697Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-12-04T09:20:54.6295823Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-12-04T09:20:54.6295954Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-12-04T09:20:54.6296091Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-12-04T09:20:54.6296233Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-12-04T09:20:54.6296829Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-12-04T09:20:54.6301057Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-12-04T09:20:54.6305394Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-12-04T09:20:54.6310453Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-12-04T09:20:54.6312690Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-12-04T09:20:54.6312967Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-12-04T09:20:54.6313230Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-12-04T09:20:54.6313457Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-12-04T09:20:54.6313711Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-12-04T09:20:54.6313852Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-12-04T09:20:54.6314018Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-12-04T09:20:54.6314215Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-12-04T09:20:54.6314471Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T09:20:54.6314718Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T09:20:54.6314846Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-12-04T09:20:54.6314972Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-12-04T09:20:54.6315094Z * [new branch] xmfan/test -> origin/xmfan/test 2025-12-04T09:20:54.6315247Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-12-04T09:20:54.6315398Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-12-04T09:20:54.6315555Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-12-04T09:20:54.6315677Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-12-04T09:20:54.6315851Z * [new branch] yiming/run_with_start_end_rng_hop -> origin/yiming/run_with_start_end_rng_hop 2025-12-04T09:20:54.6315969Z * [new branch] yolo-llama3 -> origin/yolo-llama3 2025-12-04T09:20:54.6316113Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-12-04T09:20:54.6316405Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-12-04T09:20:54.6316551Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-12-04T09:20:54.6316668Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-12-04T09:20:54.6316841Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-12-04T09:20:54.6316957Z * [new branch] zb2p -> origin/zb2p 2025-12-04T09:20:54.6317108Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-12-04T09:20:54.6317257Z * [new branch] zhxchen17/ci/vllm_lora_oom -> origin/zhxchen17/ci/vllm_lora_oom 2025-12-04T09:20:54.6321890Z * [new branch] zhxchen17/ci/vllm_multimodal_oom -> origin/zhxchen17/ci/vllm_multimodal_oom 2025-12-04T09:20:54.6324080Z * [new branch] zhxchen17/ci/vllm_pin -> origin/zhxchen17/ci/vllm_pin 2025-12-04T09:20:54.6324536Z * [new branch] zhxchen17/dynamo/unsafe_drop_all_guards -> origin/zhxchen17/dynamo/unsafe_drop_all_guards 2025-12-04T09:20:54.6324835Z * [new branch] zhxchen17/export/call_override -> origin/zhxchen17/export/call_override 2025-12-04T09:20:54.6325158Z * [new branch] zhxchen17/export/codemod1 -> origin/zhxchen17/export/codemod1 2025-12-04T09:20:54.6325365Z * [new branch] zhxchen17/export/ctx_return -> origin/zhxchen17/export/ctx_return 2025-12-04T09:20:54.6325746Z * [new branch] zhxchen17/export/disable_side_effect_warn -> origin/zhxchen17/export/disable_side_effect_warn 2025-12-04T09:20:54.6326332Z * [new branch] zhxchen17/export/pytree_check -> origin/zhxchen17/export/pytree_check 2025-12-04T09:20:54.6326535Z * [new branch] zhxchen17/precompile/aoti -> origin/zhxchen17/precompile/aoti 2025-12-04T09:20:54.6326727Z * [new branch] zhxchen17/precompile/globals -> origin/zhxchen17/precompile/globals 2025-12-04T09:20:54.6326931Z * [new branch] zhxchen17/precompile/inductor_guards -> origin/zhxchen17/precompile/inductor_guards 2025-12-04T09:20:54.6327086Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-12-04T09:20:54.6327294Z * [new branch] zhxchen17/torch_export_api_update -> origin/zhxchen17/torch_export_api_update 2025-12-04T09:20:54.6327448Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-12-04T09:20:54.6327588Z * [new branch] zxiiro/build-times -> origin/zxiiro/build-times 2025-12-04T09:20:54.6327890Z * [new branch] zxiiro/c7i.2xlarge -> origin/zxiiro/c7i.2xlarge 2025-12-04T09:20:54.6328116Z * [new branch] zxiiro/c7i.2xlarge.h100 -> origin/zxiiro/c7i.2xlarge.h100 2025-12-04T09:20:54.6328244Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-12-04T09:20:54.6328734Z * [new branch] zxiiro/risc64 -> origin/zxiiro/risc64 2025-12-04T09:20:54.6329221Z * [new branch] zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc 2025-12-04T09:20:54.6331128Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-12-04T09:20:54.6331298Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-12-04T09:20:54.6332007Z * [new tag] ciflow/b200/115316 -> ciflow/b200/115316 2025-12-04T09:20:54.6338713Z * [new tag] ciflow/b200/160685 -> ciflow/b200/160685 2025-12-04T09:20:54.6340492Z * [new tag] ciflow/b200/161607 -> ciflow/b200/161607 2025-12-04T09:20:54.6340619Z * [new tag] ciflow/b200/161938 -> ciflow/b200/161938 2025-12-04T09:20:54.6340978Z * [new tag] ciflow/b200/167207 -> ciflow/b200/167207 2025-12-04T09:20:54.6341086Z * [new tag] ciflow/b200/167989 -> ciflow/b200/167989 2025-12-04T09:20:54.6341190Z * [new tag] ciflow/b200/168096 -> ciflow/b200/168096 2025-12-04T09:20:54.6341302Z * [new tag] ciflow/b200/168175 -> ciflow/b200/168175 2025-12-04T09:20:54.6341452Z * [new tag] ciflow/b200/168195 -> ciflow/b200/168195 2025-12-04T09:20:54.6341566Z * [new tag] ciflow/b200/169200 -> ciflow/b200/169200 2025-12-04T09:20:54.6341672Z * [new tag] ciflow/b200/169216 -> ciflow/b200/169216 2025-12-04T09:20:54.6341775Z * [new tag] ciflow/b200/169380 -> ciflow/b200/169380 2025-12-04T09:20:54.6347455Z * [new tag] ciflow/b200/169412 -> ciflow/b200/169412 2025-12-04T09:20:54.6352222Z * [new tag] ciflow/b200/169470 -> ciflow/b200/169470 2025-12-04T09:20:54.6356500Z * [new tag] ciflow/b200/169471 -> ciflow/b200/169471 2025-12-04T09:20:54.6358329Z * [new tag] ciflow/b200/169472 -> ciflow/b200/169472 2025-12-04T09:20:54.6358511Z * [new tag] ciflow/b200/169514 -> ciflow/b200/169514 2025-12-04T09:20:54.6358651Z * [new tag] ciflow/b200/169517 -> ciflow/b200/169517 2025-12-04T09:20:54.6358860Z * [new tag] ciflow/binaries/165922 -> ciflow/binaries/165922 2025-12-04T09:20:54.6359000Z * [new tag] ciflow/binaries/169510 -> ciflow/binaries/169510 2025-12-04T09:20:54.6359223Z * [new tag] ciflow/binaries_wheel/157994 -> ciflow/binaries_wheel/157994 2025-12-04T09:20:54.6359371Z * [new tag] ciflow/binaries_wheel/166829 -> ciflow/binaries_wheel/166829 2025-12-04T09:20:54.6359586Z * [new tag] ciflow/binaries_wheel/167972 -> ciflow/binaries_wheel/167972 2025-12-04T09:20:54.6359921Z * [new tag] ciflow/binaries_wheel/167981 -> ciflow/binaries_wheel/167981 2025-12-04T09:20:54.6360063Z * [new tag] ciflow/dynamo/167695 -> ciflow/dynamo/167695 2025-12-04T09:20:54.6360176Z * [new tag] ciflow/dynamo/168096 -> ciflow/dynamo/168096 2025-12-04T09:20:54.6360360Z * [new tag] ciflow/dynamo/169525 -> ciflow/dynamo/169525 2025-12-04T09:20:54.6365789Z * [new tag] ciflow/h100-cutlass-backend/161938 -> ciflow/h100-cutlass-backend/161938 2025-12-04T09:20:54.6366129Z * [new tag] ciflow/h100-cutlass-backend/161940 -> ciflow/h100-cutlass-backend/161940 2025-12-04T09:20:54.6366344Z * [new tag] ciflow/h100-distributed/168923 -> ciflow/h100-distributed/168923 2025-12-04T09:20:54.6366482Z * [new tag] ciflow/h100-symm-mem/167552 -> ciflow/h100-symm-mem/167552 2025-12-04T09:20:54.6366614Z * [new tag] ciflow/h100-symm-mem/168129 -> ciflow/h100-symm-mem/168129 2025-12-04T09:20:54.6366748Z * [new tag] ciflow/h100-symm-mem/168917 -> ciflow/h100-symm-mem/168917 2025-12-04T09:20:54.6366873Z * [new tag] ciflow/h100-symm-mem/169156 -> ciflow/h100-symm-mem/169156 2025-12-04T09:20:54.6366991Z * [new tag] ciflow/h100-symm-mem/169200 -> ciflow/h100-symm-mem/169200 2025-12-04T09:20:54.6367117Z * [new tag] ciflow/h100-symm-mem/169216 -> ciflow/h100-symm-mem/169216 2025-12-04T09:20:54.6367245Z * [new tag] ciflow/h100-symm-mem/169338 -> ciflow/h100-symm-mem/169338 2025-12-04T09:20:54.6367364Z * [new tag] ciflow/h100-symm-mem/169355 -> ciflow/h100-symm-mem/169355 2025-12-04T09:20:54.6367483Z * [new tag] ciflow/h100-symm-mem/169543 -> ciflow/h100-symm-mem/169543 2025-12-04T09:20:54.6367599Z * [new tag] ciflow/h100/115316 -> ciflow/h100/115316 2025-12-04T09:20:54.6367705Z * [new tag] ciflow/h100/160685 -> ciflow/h100/160685 2025-12-04T09:20:54.6367956Z * [new tag] ciflow/h100/160729 -> ciflow/h100/160729 2025-12-04T09:20:54.6368063Z * [new tag] ciflow/h100/161607 -> ciflow/h100/161607 2025-12-04T09:20:54.6368169Z * [new tag] ciflow/h100/161938 -> ciflow/h100/161938 2025-12-04T09:20:54.6368328Z * [new tag] ciflow/h100/167207 -> ciflow/h100/167207 2025-12-04T09:20:54.6368430Z * [new tag] ciflow/h100/167989 -> ciflow/h100/167989 2025-12-04T09:20:54.6368539Z * [new tag] ciflow/h100/168096 -> ciflow/h100/168096 2025-12-04T09:20:54.6368640Z * [new tag] ciflow/h100/168175 -> ciflow/h100/168175 2025-12-04T09:20:54.6368740Z * [new tag] ciflow/h100/168195 -> ciflow/h100/168195 2025-12-04T09:20:54.6368846Z * [new tag] ciflow/h100/168980 -> ciflow/h100/168980 2025-12-04T09:20:54.6368948Z * [new tag] ciflow/h100/169200 -> ciflow/h100/169200 2025-12-04T09:20:54.6369048Z * [new tag] ciflow/h100/169216 -> ciflow/h100/169216 2025-12-04T09:20:54.6369156Z * [new tag] ciflow/h100/169380 -> ciflow/h100/169380 2025-12-04T09:20:54.6369257Z * [new tag] ciflow/h100/169412 -> ciflow/h100/169412 2025-12-04T09:20:54.6369364Z * [new tag] ciflow/h100/169470 -> ciflow/h100/169470 2025-12-04T09:20:54.6369463Z * [new tag] ciflow/h100/169471 -> ciflow/h100/169471 2025-12-04T09:20:54.6369580Z * [new tag] ciflow/h100/169472 -> ciflow/h100/169472 2025-12-04T09:20:54.6369690Z * [new tag] ciflow/h100/169514 -> ciflow/h100/169514 2025-12-04T09:20:54.6369834Z * [new tag] ciflow/inductor-cu126/168096 -> ciflow/inductor-cu126/168096 2025-12-04T09:20:54.6370091Z * [new tag] ciflow/inductor-micro-benchmark-cpu-x86/168096 -> ciflow/inductor-micro-benchmark-cpu-x86/168096 2025-12-04T09:20:54.6370281Z * [new tag] ciflow/inductor-micro-benchmark/166165 -> ciflow/inductor-micro-benchmark/166165 2025-12-04T09:20:54.6370468Z * [new tag] ciflow/inductor-micro-benchmark/168096 -> ciflow/inductor-micro-benchmark/168096 2025-12-04T09:20:54.6370662Z * [new tag] ciflow/inductor-perf-compare/168096 -> ciflow/inductor-perf-compare/168096 2025-12-04T09:20:54.6370932Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168073 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168073 2025-12-04T09:20:54.6371198Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168096 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168096 2025-12-04T09:20:54.6371546Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi300/169024 2025-12-04T09:20:54.6371856Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi355/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi355/169024 2025-12-04T09:20:54.6372091Z * [new tag] ciflow/inductor-perf-test-nightly/168096 -> ciflow/inductor-perf-test-nightly/168096 2025-12-04T09:20:54.6372264Z * [new tag] ciflow/inductor-periodic/168096 -> ciflow/inductor-periodic/168096 2025-12-04T09:20:54.6372585Z * [new tag] ciflow/inductor-periodic/169024 -> ciflow/inductor-periodic/169024 2025-12-04T09:20:54.6372762Z * [new tag] ciflow/inductor-periodic/169425 -> ciflow/inductor-periodic/169425 2025-12-04T09:20:54.6372938Z * [new tag] ciflow/inductor-rocm-mi200/165545 -> ciflow/inductor-rocm-mi200/165545 2025-12-04T09:20:54.6373121Z * [new tag] ciflow/inductor-rocm-mi200/165997 -> ciflow/inductor-rocm-mi200/165997 2025-12-04T09:20:54.6373271Z * [new tag] ciflow/inductor-rocm-mi200/168096 -> ciflow/inductor-rocm-mi200/168096 2025-12-04T09:20:54.6373528Z * [new tag] ciflow/inductor-rocm-mi200/169063 -> ciflow/inductor-rocm-mi200/169063 2025-12-04T09:20:54.6373678Z * [new tag] ciflow/inductor-rocm-mi200/169425 -> ciflow/inductor-rocm-mi200/169425 2025-12-04T09:20:54.6373883Z * [new tag] ciflow/inductor-rocm-mi300/165545 -> ciflow/inductor-rocm-mi300/165545 2025-12-04T09:20:54.6374044Z * [new tag] ciflow/inductor-rocm-mi300/168096 -> ciflow/inductor-rocm-mi300/168096 2025-12-04T09:20:54.6374194Z * [new tag] ciflow/inductor-rocm-mi300/169063 -> ciflow/inductor-rocm-mi300/169063 2025-12-04T09:20:54.6374350Z * [new tag] ciflow/inductor-rocm-mi300/169425 -> ciflow/inductor-rocm-mi300/169425 2025-12-04T09:20:54.6374497Z * [new tag] ciflow/inductor-rocm/162052 -> ciflow/inductor-rocm/162052 2025-12-04T09:20:54.6374632Z * [new tag] ciflow/inductor-rocm/168971 -> ciflow/inductor-rocm/168971 2025-12-04T09:20:54.6374793Z * [new tag] ciflow/inductor-windows/168096 -> ciflow/inductor-windows/168096 2025-12-04T09:20:54.6374917Z * [new tag] ciflow/inductor/144542 -> ciflow/inductor/144542 2025-12-04T09:20:54.6375047Z * [new tag] ciflow/inductor/146506 -> ciflow/inductor/146506 2025-12-04T09:20:54.6375164Z * [new tag] ciflow/inductor/147990 -> ciflow/inductor/147990 2025-12-04T09:20:54.6375280Z * [new tag] ciflow/inductor/148294 -> ciflow/inductor/148294 2025-12-04T09:20:54.6375403Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-12-04T09:20:54.6375515Z * [new tag] ciflow/inductor/157149 -> ciflow/inductor/157149 2025-12-04T09:20:54.6375637Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-12-04T09:20:54.6375754Z * [new tag] ciflow/inductor/160685 -> ciflow/inductor/160685 2025-12-04T09:20:54.6375869Z * [new tag] ciflow/inductor/160686 -> ciflow/inductor/160686 2025-12-04T09:20:54.6375991Z * [new tag] ciflow/inductor/160687 -> ciflow/inductor/160687 2025-12-04T09:20:54.6376228Z * [new tag] ciflow/inductor/160688 -> ciflow/inductor/160688 2025-12-04T09:20:54.6376682Z * [new tag] ciflow/inductor/160706 -> ciflow/inductor/160706 2025-12-04T09:20:54.6378043Z * [new tag] ciflow/inductor/160729 -> ciflow/inductor/160729 2025-12-04T09:20:54.6378379Z * [new tag] ciflow/inductor/161938 -> ciflow/inductor/161938 2025-12-04T09:20:54.6378559Z * [new tag] ciflow/inductor/161939 -> ciflow/inductor/161939 2025-12-04T09:20:54.6378701Z * [new tag] ciflow/inductor/161940 -> ciflow/inductor/161940 2025-12-04T09:20:54.6381099Z * [new tag] ciflow/inductor/162052 -> ciflow/inductor/162052 2025-12-04T09:20:54.6381433Z * [new tag] ciflow/inductor/162275 -> ciflow/inductor/162275 2025-12-04T09:20:54.6381623Z * [new tag] ciflow/inductor/162795 -> ciflow/inductor/162795 2025-12-04T09:20:54.6381775Z * [new tag] ciflow/inductor/163245 -> ciflow/inductor/163245 2025-12-04T09:20:54.6381919Z * [new tag] ciflow/inductor/163335 -> ciflow/inductor/163335 2025-12-04T09:20:54.6382044Z * [new tag] ciflow/inductor/163503 -> ciflow/inductor/163503 2025-12-04T09:20:54.6382284Z * [new tag] ciflow/inductor/163942 -> ciflow/inductor/163942 2025-12-04T09:20:54.6384776Z * [new tag] ciflow/inductor/165270 -> ciflow/inductor/165270 2025-12-04T09:20:54.6389241Z * [new tag] ciflow/inductor/165274 -> ciflow/inductor/165274 2025-12-04T09:20:54.6391717Z * [new tag] ciflow/inductor/165322 -> ciflow/inductor/165322 2025-12-04T09:20:54.6396411Z * [new tag] ciflow/inductor/165597 -> ciflow/inductor/165597 2025-12-04T09:20:54.6401548Z * [new tag] ciflow/inductor/166063 -> ciflow/inductor/166063 2025-12-04T09:20:54.6405719Z * [new tag] ciflow/inductor/166075 -> ciflow/inductor/166075 2025-12-04T09:20:54.6408470Z * [new tag] ciflow/inductor/166165 -> ciflow/inductor/166165 2025-12-04T09:20:54.6408678Z * [new tag] ciflow/inductor/166254 -> ciflow/inductor/166254 2025-12-04T09:20:54.6408801Z * [new tag] ciflow/inductor/166483 -> ciflow/inductor/166483 2025-12-04T09:20:54.6408922Z * [new tag] ciflow/inductor/166494 -> ciflow/inductor/166494 2025-12-04T09:20:54.6409049Z * [new tag] ciflow/inductor/166545 -> ciflow/inductor/166545 2025-12-04T09:20:54.6409166Z * [new tag] ciflow/inductor/166788 -> ciflow/inductor/166788 2025-12-04T09:20:54.6409300Z * [new tag] ciflow/inductor/166846 -> ciflow/inductor/166846 2025-12-04T09:20:54.6409415Z * [new tag] ciflow/inductor/167300 -> ciflow/inductor/167300 2025-12-04T09:20:54.6409532Z * [new tag] ciflow/inductor/167407 -> ciflow/inductor/167407 2025-12-04T09:20:54.6409660Z * [new tag] ciflow/inductor/167536 -> ciflow/inductor/167536 2025-12-04T09:20:54.6409777Z * [new tag] ciflow/inductor/167552 -> ciflow/inductor/167552 2025-12-04T09:20:54.6409892Z * [new tag] ciflow/inductor/167555 -> ciflow/inductor/167555 2025-12-04T09:20:54.6410016Z * [new tag] ciflow/inductor/167583 -> ciflow/inductor/167583 2025-12-04T09:20:54.6410130Z * [new tag] ciflow/inductor/167599 -> ciflow/inductor/167599 2025-12-04T09:20:54.6410250Z * [new tag] ciflow/inductor/167647 -> ciflow/inductor/167647 2025-12-04T09:20:54.6410371Z * [new tag] ciflow/inductor/167677 -> ciflow/inductor/167677 2025-12-04T09:20:54.6410508Z * [new tag] ciflow/inductor/167680 -> ciflow/inductor/167680 2025-12-04T09:20:54.6410629Z * [new tag] ciflow/inductor/167695 -> ciflow/inductor/167695 2025-12-04T09:20:54.6410744Z * [new tag] ciflow/inductor/167742 -> ciflow/inductor/167742 2025-12-04T09:20:54.6410868Z * [new tag] ciflow/inductor/167768 -> ciflow/inductor/167768 2025-12-04T09:20:54.6410984Z * [new tag] ciflow/inductor/167773 -> ciflow/inductor/167773 2025-12-04T09:20:54.6411099Z * [new tag] ciflow/inductor/167781 -> ciflow/inductor/167781 2025-12-04T09:20:54.6411230Z * [new tag] ciflow/inductor/167880 -> ciflow/inductor/167880 2025-12-04T09:20:54.6411344Z * [new tag] ciflow/inductor/167887 -> ciflow/inductor/167887 2025-12-04T09:20:54.6411655Z * [new tag] ciflow/inductor/167972 -> ciflow/inductor/167972 2025-12-04T09:20:54.6411809Z * [new tag] ciflow/inductor/167989 -> ciflow/inductor/167989 2025-12-04T09:20:54.6411931Z * [new tag] ciflow/inductor/168002 -> ciflow/inductor/168002 2025-12-04T09:20:54.6412053Z * [new tag] ciflow/inductor/168050 -> ciflow/inductor/168050 2025-12-04T09:20:54.6412169Z * [new tag] ciflow/inductor/168051 -> ciflow/inductor/168051 2025-12-04T09:20:54.6412280Z * [new tag] ciflow/inductor/168052 -> ciflow/inductor/168052 2025-12-04T09:20:54.6412397Z * [new tag] ciflow/inductor/168073 -> ciflow/inductor/168073 2025-12-04T09:20:54.6412508Z * [new tag] ciflow/inductor/168096 -> ciflow/inductor/168096 2025-12-04T09:20:54.6412627Z * [new tag] ciflow/inductor/168114 -> ciflow/inductor/168114 2025-12-04T09:20:54.6412740Z * [new tag] ciflow/inductor/168115 -> ciflow/inductor/168115 2025-12-04T09:20:54.6412902Z * [new tag] ciflow/inductor/168127 -> ciflow/inductor/168127 2025-12-04T09:20:54.6413024Z * [new tag] ciflow/inductor/168129 -> ciflow/inductor/168129 2025-12-04T09:20:54.6413138Z * [new tag] ciflow/inductor/168157 -> ciflow/inductor/168157 2025-12-04T09:20:54.6413298Z * [new tag] ciflow/inductor/168175 -> ciflow/inductor/168175 2025-12-04T09:20:54.6413413Z * [new tag] ciflow/inductor/168185 -> ciflow/inductor/168185 2025-12-04T09:20:54.6413527Z * [new tag] ciflow/inductor/168195 -> ciflow/inductor/168195 2025-12-04T09:20:54.6413649Z * [new tag] ciflow/inductor/168209 -> ciflow/inductor/168209 2025-12-04T09:20:54.6413762Z * [new tag] ciflow/inductor/168266 -> ciflow/inductor/168266 2025-12-04T09:20:54.6413874Z * [new tag] ciflow/inductor/168316 -> ciflow/inductor/168316 2025-12-04T09:20:54.6413998Z * [new tag] ciflow/inductor/168326 -> ciflow/inductor/168326 2025-12-04T09:20:54.6414109Z * [new tag] ciflow/inductor/168368 -> ciflow/inductor/168368 2025-12-04T09:20:54.6414228Z * [new tag] ciflow/inductor/168894 -> ciflow/inductor/168894 2025-12-04T09:20:54.6414347Z * [new tag] ciflow/inductor/168934 -> ciflow/inductor/168934 2025-12-04T09:20:54.6414459Z * [new tag] ciflow/inductor/168939 -> ciflow/inductor/168939 2025-12-04T09:20:54.6414587Z * [new tag] ciflow/inductor/168946 -> ciflow/inductor/168946 2025-12-04T09:20:54.6414698Z * [new tag] ciflow/inductor/168950 -> ciflow/inductor/168950 2025-12-04T09:20:54.6414819Z * [new tag] ciflow/inductor/168951 -> ciflow/inductor/168951 2025-12-04T09:20:54.6414933Z * [new tag] ciflow/inductor/168952 -> ciflow/inductor/168952 2025-12-04T09:20:54.6415048Z * [new tag] ciflow/inductor/168955 -> ciflow/inductor/168955 2025-12-04T09:20:54.6415173Z * [new tag] ciflow/inductor/168971 -> ciflow/inductor/168971 2025-12-04T09:20:54.6415287Z * [new tag] ciflow/inductor/168979 -> ciflow/inductor/168979 2025-12-04T09:20:54.6415410Z * [new tag] ciflow/inductor/168980 -> ciflow/inductor/168980 2025-12-04T09:20:54.6415527Z * [new tag] ciflow/inductor/168983 -> ciflow/inductor/168983 2025-12-04T09:20:54.6415642Z * [new tag] ciflow/inductor/169006 -> ciflow/inductor/169006 2025-12-04T09:20:54.6415761Z * [new tag] ciflow/inductor/169023 -> ciflow/inductor/169023 2025-12-04T09:20:54.6415874Z * [new tag] ciflow/inductor/169024 -> ciflow/inductor/169024 2025-12-04T09:20:54.6416000Z * [new tag] ciflow/inductor/169025 -> ciflow/inductor/169025 2025-12-04T09:20:54.6416116Z * [new tag] ciflow/inductor/169066 -> ciflow/inductor/169066 2025-12-04T09:20:54.6416229Z * [new tag] ciflow/inductor/169091 -> ciflow/inductor/169091 2025-12-04T09:20:54.6416543Z * [new tag] ciflow/inductor/169102 -> ciflow/inductor/169102 2025-12-04T09:20:54.6417070Z * [new tag] ciflow/inductor/169103 -> ciflow/inductor/169103 2025-12-04T09:20:54.6417346Z * [new tag] ciflow/inductor/169121 -> ciflow/inductor/169121 2025-12-04T09:20:54.6417490Z * [new tag] ciflow/inductor/169134 -> ciflow/inductor/169134 2025-12-04T09:20:54.6419382Z * [new tag] ciflow/inductor/169135 -> ciflow/inductor/169135 2025-12-04T09:20:54.6419691Z * [new tag] ciflow/inductor/169141 -> ciflow/inductor/169141 2025-12-04T09:20:54.6419866Z * [new tag] ciflow/inductor/169151 -> ciflow/inductor/169151 2025-12-04T09:20:54.6420263Z * [new tag] ciflow/inductor/169161 -> ciflow/inductor/169161 2025-12-04T09:20:54.6420398Z * [new tag] ciflow/inductor/169167 -> ciflow/inductor/169167 2025-12-04T09:20:54.6420597Z * [new tag] ciflow/inductor/169177 -> ciflow/inductor/169177 2025-12-04T09:20:54.6421957Z * [new tag] ciflow/inductor/169185 -> ciflow/inductor/169185 2025-12-04T09:20:54.6422294Z * [new tag] ciflow/inductor/169196 -> ciflow/inductor/169196 2025-12-04T09:20:54.6422511Z * [new tag] ciflow/inductor/169200 -> ciflow/inductor/169200 2025-12-04T09:20:54.6422671Z * [new tag] ciflow/inductor/169204 -> ciflow/inductor/169204 2025-12-04T09:20:54.6423052Z * [new tag] ciflow/inductor/169216 -> ciflow/inductor/169216 2025-12-04T09:20:54.6423489Z * [new tag] ciflow/inductor/169219 -> ciflow/inductor/169219 2025-12-04T09:20:54.6423925Z * [new tag] ciflow/inductor/169220 -> ciflow/inductor/169220 2025-12-04T09:20:54.6424630Z * [new tag] ciflow/inductor/169230 -> ciflow/inductor/169230 2025-12-04T09:20:54.6424953Z * [new tag] ciflow/inductor/169242 -> ciflow/inductor/169242 2025-12-04T09:20:54.6427594Z * [new tag] ciflow/inductor/169245 -> ciflow/inductor/169245 2025-12-04T09:20:54.6427918Z * [new tag] ciflow/inductor/169260 -> ciflow/inductor/169260 2025-12-04T09:20:54.6428088Z * [new tag] ciflow/inductor/169282 -> ciflow/inductor/169282 2025-12-04T09:20:54.6428235Z * [new tag] ciflow/inductor/169286 -> ciflow/inductor/169286 2025-12-04T09:20:54.6428373Z * [new tag] ciflow/inductor/169299 -> ciflow/inductor/169299 2025-12-04T09:20:54.6428565Z * [new tag] ciflow/inductor/169304 -> ciflow/inductor/169304 2025-12-04T09:20:54.6428929Z * [new tag] ciflow/inductor/169305 -> ciflow/inductor/169305 2025-12-04T09:20:54.6430082Z * [new tag] ciflow/inductor/169308 -> ciflow/inductor/169308 2025-12-04T09:20:54.6430383Z * [new tag] ciflow/inductor/169319 -> ciflow/inductor/169319 2025-12-04T09:20:54.6430522Z * [new tag] ciflow/inductor/169326 -> ciflow/inductor/169326 2025-12-04T09:20:54.6430818Z * [new tag] ciflow/inductor/169332 -> ciflow/inductor/169332 2025-12-04T09:20:54.6433086Z * [new tag] ciflow/inductor/169333 -> ciflow/inductor/169333 2025-12-04T09:20:54.6433248Z * [new tag] ciflow/inductor/169336 -> ciflow/inductor/169336 2025-12-04T09:20:54.6433374Z * [new tag] ciflow/inductor/169340 -> ciflow/inductor/169340 2025-12-04T09:20:54.6433498Z * [new tag] ciflow/inductor/169341 -> ciflow/inductor/169341 2025-12-04T09:20:54.6433785Z * [new tag] ciflow/inductor/169343 -> ciflow/inductor/169343 2025-12-04T09:20:54.6433940Z * [new tag] ciflow/inductor/169346 -> ciflow/inductor/169346 2025-12-04T09:20:54.6434360Z * [new tag] ciflow/inductor/169348 -> ciflow/inductor/169348 2025-12-04T09:20:54.6435537Z * [new tag] ciflow/inductor/169350 -> ciflow/inductor/169350 2025-12-04T09:20:54.6435892Z * [new tag] ciflow/inductor/169355 -> ciflow/inductor/169355 2025-12-04T09:20:54.6436034Z * [new tag] ciflow/inductor/169370 -> ciflow/inductor/169370 2025-12-04T09:20:54.6438438Z * [new tag] ciflow/inductor/169375 -> ciflow/inductor/169375 2025-12-04T09:20:54.6438729Z * [new tag] ciflow/inductor/169389 -> ciflow/inductor/169389 2025-12-04T09:20:54.6438889Z * [new tag] ciflow/inductor/169391 -> ciflow/inductor/169391 2025-12-04T09:20:54.6439003Z * [new tag] ciflow/inductor/169393 -> ciflow/inductor/169393 2025-12-04T09:20:54.6439268Z * [new tag] ciflow/inductor/169399 -> ciflow/inductor/169399 2025-12-04T09:20:54.6439522Z * [new tag] ciflow/inductor/169400 -> ciflow/inductor/169400 2025-12-04T09:20:54.6439670Z * [new tag] ciflow/inductor/169415 -> ciflow/inductor/169415 2025-12-04T09:20:54.6440134Z * [new tag] ciflow/inductor/169417 -> ciflow/inductor/169417 2025-12-04T09:20:54.6440508Z * [new tag] ciflow/inductor/169418 -> ciflow/inductor/169418 2025-12-04T09:20:54.6445148Z * [new tag] ciflow/inductor/169430 -> ciflow/inductor/169430 2025-12-04T09:20:54.6445319Z * [new tag] ciflow/inductor/169432 -> ciflow/inductor/169432 2025-12-04T09:20:54.6445444Z * [new tag] ciflow/inductor/169436 -> ciflow/inductor/169436 2025-12-04T09:20:54.6445571Z * [new tag] ciflow/inductor/169437 -> ciflow/inductor/169437 2025-12-04T09:20:54.6445705Z * [new tag] ciflow/inductor/169438 -> ciflow/inductor/169438 2025-12-04T09:20:54.6445822Z * [new tag] ciflow/inductor/169441 -> ciflow/inductor/169441 2025-12-04T09:20:54.6445955Z * [new tag] ciflow/inductor/169446 -> ciflow/inductor/169446 2025-12-04T09:20:54.6446085Z * [new tag] ciflow/inductor/169447 -> ciflow/inductor/169447 2025-12-04T09:20:54.6446207Z * [new tag] ciflow/inductor/169452 -> ciflow/inductor/169452 2025-12-04T09:20:54.6446326Z * [new tag] ciflow/inductor/169455 -> ciflow/inductor/169455 2025-12-04T09:20:54.6446438Z * [new tag] ciflow/inductor/169459 -> ciflow/inductor/169459 2025-12-04T09:20:54.6446565Z * [new tag] ciflow/inductor/169463 -> ciflow/inductor/169463 2025-12-04T09:20:54.6447151Z * [new tag] ciflow/inductor/169476 -> ciflow/inductor/169476 2025-12-04T09:20:54.6447614Z * [new tag] ciflow/inductor/169485 -> ciflow/inductor/169485 2025-12-04T09:20:54.6448026Z * [new tag] ciflow/inductor/169493 -> ciflow/inductor/169493 2025-12-04T09:20:54.6448445Z * [new tag] ciflow/inductor/169496 -> ciflow/inductor/169496 2025-12-04T09:20:54.6448933Z * [new tag] ciflow/inductor/169497 -> ciflow/inductor/169497 2025-12-04T09:20:54.6449538Z * [new tag] ciflow/inductor/169503 -> ciflow/inductor/169503 2025-12-04T09:20:54.6449942Z * [new tag] ciflow/inductor/169504 -> ciflow/inductor/169504 2025-12-04T09:20:54.6450799Z * [new tag] ciflow/inductor/169505 -> ciflow/inductor/169505 2025-12-04T09:20:54.6451399Z * [new tag] ciflow/inductor/169508 -> ciflow/inductor/169508 2025-12-04T09:20:54.6451888Z * [new tag] ciflow/inductor/169509 -> ciflow/inductor/169509 2025-12-04T09:20:54.6452708Z * [new tag] ciflow/inductor/169513 -> ciflow/inductor/169513 2025-12-04T09:20:54.6452835Z * [new tag] ciflow/inductor/169514 -> ciflow/inductor/169514 2025-12-04T09:20:54.6455269Z * [new tag] ciflow/inductor/169515 -> ciflow/inductor/169515 2025-12-04T09:20:54.6455447Z * [new tag] ciflow/inductor/169517 -> ciflow/inductor/169517 2025-12-04T09:20:54.6455561Z * [new tag] ciflow/inductor/169519 -> ciflow/inductor/169519 2025-12-04T09:20:54.6455678Z * [new tag] ciflow/inductor/169520 -> ciflow/inductor/169520 2025-12-04T09:20:54.6455791Z * [new tag] ciflow/inductor/169521 -> ciflow/inductor/169521 2025-12-04T09:20:54.6456058Z * [new tag] ciflow/inductor/169524 -> ciflow/inductor/169524 2025-12-04T09:20:54.6456253Z * [new tag] ciflow/inductor/169527 -> ciflow/inductor/169527 2025-12-04T09:20:54.6456697Z * [new tag] ciflow/inductor/169528 -> ciflow/inductor/169528 2025-12-04T09:20:54.6458313Z * [new tag] ciflow/inductor/169532 -> ciflow/inductor/169532 2025-12-04T09:20:54.6458619Z * [new tag] ciflow/inductor/169535 -> ciflow/inductor/169535 2025-12-04T09:20:54.6458943Z * [new tag] ciflow/inductor/169536 -> ciflow/inductor/169536 2025-12-04T09:20:54.6459258Z * [new tag] ciflow/inductor/169547 -> ciflow/inductor/169547 2025-12-04T09:20:54.6459379Z * [new tag] ciflow/inductor/169548 -> ciflow/inductor/169548 2025-12-04T09:20:54.6459543Z * [new tag] ciflow/inductor/169549 -> ciflow/inductor/169549 2025-12-04T09:20:54.6459958Z * [new tag] ciflow/inductor/169551 -> ciflow/inductor/169551 2025-12-04T09:20:54.6461922Z * [new tag] ciflow/inductor/169552 -> ciflow/inductor/169552 2025-12-04T09:20:54.6462075Z * [new tag] ciflow/inductor/169553 -> ciflow/inductor/169553 2025-12-04T09:20:54.6462214Z * [new tag] ciflow/inductor/169557 -> ciflow/inductor/169557 2025-12-04T09:20:54.6462378Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-12-04T09:20:54.6463983Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-12-04T09:20:54.6464327Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-12-04T09:20:54.6464486Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-12-04T09:20:54.6464656Z * [new tag] ciflow/linux-aarch64/166075 -> ciflow/linux-aarch64/166075 2025-12-04T09:20:54.6464943Z * [new tag] ciflow/linux-aarch64/166876 -> ciflow/linux-aarch64/166876 2025-12-04T09:20:54.6466414Z * [new tag] ciflow/linux-aarch64/167981 -> ciflow/linux-aarch64/167981 2025-12-04T09:20:54.6466581Z * [new tag] ciflow/mps/166254 -> ciflow/mps/166254 2025-12-04T09:20:54.6466694Z * [new tag] ciflow/mps/169017 -> ciflow/mps/169017 2025-12-04T09:20:54.6467056Z * [new tag] ciflow/mps/169372 -> ciflow/mps/169372 2025-12-04T09:20:54.6467405Z * [new tag] ciflow/mps/169478 -> ciflow/mps/169478 2025-12-04T09:20:54.6468954Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-12-04T09:20:54.6469134Z * [new tag] ciflow/op-benchmark/166075 -> ciflow/op-benchmark/166075 2025-12-04T09:20:54.6469427Z * [new tag] ciflow/op-benchmark/169544 -> ciflow/op-benchmark/169544 2025-12-04T09:20:54.6469617Z * [new tag] ciflow/periodic-rocm-mi200/165997 -> ciflow/periodic-rocm-mi200/165997 2025-12-04T09:20:54.6470040Z * [new tag] ciflow/periodic-rocm-mi200/166517 -> ciflow/periodic-rocm-mi200/166517 2025-12-04T09:20:54.6470475Z * [new tag] ciflow/periodic-rocm-mi200/169063 -> ciflow/periodic-rocm-mi200/169063 2025-12-04T09:20:54.6470970Z * [new tag] ciflow/periodic-rocm-mi200/169425 -> ciflow/periodic-rocm-mi200/169425 2025-12-04T09:20:54.6471427Z * [new tag] ciflow/periodic-rocm-mi300/166517 -> ciflow/periodic-rocm-mi300/166517 2025-12-04T09:20:54.6471855Z * [new tag] ciflow/periodic-rocm-mi300/169063 -> ciflow/periodic-rocm-mi300/169063 2025-12-04T09:20:54.6472229Z * [new tag] ciflow/periodic-rocm-mi300/169425 -> ciflow/periodic-rocm-mi300/169425 2025-12-04T09:20:54.6473621Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-12-04T09:20:54.6473886Z * [new tag] ciflow/periodic/167207 -> ciflow/periodic/167207 2025-12-04T09:20:54.6474289Z * [new tag] ciflow/periodic/167978 -> ciflow/periodic/167978 2025-12-04T09:20:54.6474653Z * [new tag] ciflow/periodic/168096 -> ciflow/periodic/168096 2025-12-04T09:20:54.6475200Z * [new tag] ciflow/periodic/169286 -> ciflow/periodic/169286 2025-12-04T09:20:54.6475818Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-12-04T09:20:54.6476269Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-12-04T09:20:54.6476951Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-12-04T09:20:54.6477399Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-12-04T09:20:54.6478610Z * [new tag] ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T09:20:54.6478887Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-12-04T09:20:54.6479876Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-12-04T09:20:54.6480101Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-12-04T09:20:54.6481072Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-12-04T09:20:54.6481387Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-12-04T09:20:54.6482348Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-12-04T09:20:54.6482860Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-12-04T09:20:54.6483369Z * [new tag] ciflow/pull/167207 -> ciflow/pull/167207 2025-12-04T09:20:54.6484282Z * [new tag] ciflow/quantization-periodic/169207 -> ciflow/quantization-periodic/169207 2025-12-04T09:20:54.6484470Z * [new tag] ciflow/rocm-mi200/165545 -> ciflow/rocm-mi200/165545 2025-12-04T09:20:54.6484975Z * [new tag] ciflow/rocm-mi200/165997 -> ciflow/rocm-mi200/165997 2025-12-04T09:20:54.6485298Z * [new tag] ciflow/rocm-mi200/168096 -> ciflow/rocm-mi200/168096 2025-12-04T09:20:54.6486159Z * [new tag] ciflow/rocm-mi200/168275 -> ciflow/rocm-mi200/168275 2025-12-04T09:20:54.6486408Z * [new tag] ciflow/rocm-mi200/169063 -> ciflow/rocm-mi200/169063 2025-12-04T09:20:54.6486932Z * [new tag] ciflow/rocm-mi200/169356 -> ciflow/rocm-mi200/169356 2025-12-04T09:20:54.6487317Z * [new tag] ciflow/rocm-mi200/169425 -> ciflow/rocm-mi200/169425 2025-12-04T09:20:54.6487927Z * [new tag] ciflow/rocm-mi300/165545 -> ciflow/rocm-mi300/165545 2025-12-04T09:20:54.6488414Z * [new tag] ciflow/rocm-mi300/167157 -> ciflow/rocm-mi300/167157 2025-12-04T09:20:54.6488847Z * [new tag] ciflow/rocm-mi300/168096 -> ciflow/rocm-mi300/168096 2025-12-04T09:20:54.6489311Z * [new tag] ciflow/rocm-mi300/169063 -> ciflow/rocm-mi300/169063 2025-12-04T09:20:54.6489745Z * [new tag] ciflow/rocm-mi300/169425 -> ciflow/rocm-mi300/169425 2025-12-04T09:20:54.6490371Z * [new tag] ciflow/rocm-mi355/167157 -> ciflow/rocm-mi355/167157 2025-12-04T09:20:54.6490731Z * [new tag] ciflow/rocm-mi355/168275 -> ciflow/rocm-mi355/168275 2025-12-04T09:20:54.6491698Z * [new tag] ciflow/rocm-mi355/169425 -> ciflow/rocm-mi355/169425 2025-12-04T09:20:54.6491945Z * [new tag] ciflow/rocm-navi31/168275 -> ciflow/rocm-navi31/168275 2025-12-04T09:20:54.6492448Z * [new tag] ciflow/rocm-navi31/169425 -> ciflow/rocm-navi31/169425 2025-12-04T09:20:54.6493370Z * [new tag] ciflow/rocm/115316 -> ciflow/rocm/115316 2025-12-04T09:20:54.6493561Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-12-04T09:20:54.6493981Z * [new tag] ciflow/rocm/160685 -> ciflow/rocm/160685 2025-12-04T09:20:54.6494482Z * [new tag] ciflow/rocm/161607 -> ciflow/rocm/161607 2025-12-04T09:20:54.6495501Z * [new tag] ciflow/rocm/162052 -> ciflow/rocm/162052 2025-12-04T09:20:54.6495624Z * [new tag] ciflow/rocm/165997 -> ciflow/rocm/165997 2025-12-04T09:20:54.6496487Z * [new tag] ciflow/rocm/166165 -> ciflow/rocm/166165 2025-12-04T09:20:54.6496766Z * [new tag] ciflow/rocm/166517 -> ciflow/rocm/166517 2025-12-04T09:20:54.6497141Z * [new tag] ciflow/rocm/167207 -> ciflow/rocm/167207 2025-12-04T09:20:54.6497575Z * [new tag] ciflow/rocm/167536 -> ciflow/rocm/167536 2025-12-04T09:20:54.6498095Z * [new tag] ciflow/rocm/167781 -> ciflow/rocm/167781 2025-12-04T09:20:54.6498993Z * [new tag] ciflow/rocm/167989 -> ciflow/rocm/167989 2025-12-04T09:20:54.6499279Z * [new tag] ciflow/rocm/168073 -> ciflow/rocm/168073 2025-12-04T09:20:54.6500126Z * [new tag] ciflow/rocm/168195 -> ciflow/rocm/168195 2025-12-04T09:20:54.6500379Z * [new tag] ciflow/rocm/168939 -> ciflow/rocm/168939 2025-12-04T09:20:54.6500841Z * [new tag] ciflow/rocm/168971 -> ciflow/rocm/168971 2025-12-04T09:20:54.6501355Z * [new tag] ciflow/rocm/169024 -> ciflow/rocm/169024 2025-12-04T09:20:54.6501829Z * [new tag] ciflow/rocm/169200 -> ciflow/rocm/169200 2025-12-04T09:20:54.6502347Z * [new tag] ciflow/rocm/169216 -> ciflow/rocm/169216 2025-12-04T09:20:54.6503017Z * [new tag] ciflow/rocm/169312 -> ciflow/rocm/169312 2025-12-04T09:20:54.6503499Z * [new tag] ciflow/rocm/169380 -> ciflow/rocm/169380 2025-12-04T09:20:54.6503817Z * [new tag] ciflow/rocm/169427 -> ciflow/rocm/169427 2025-12-04T09:20:54.6504360Z * [new tag] ciflow/rocm/169455 -> ciflow/rocm/169455 2025-12-04T09:20:54.6504726Z * [new tag] ciflow/rocm/169470 -> ciflow/rocm/169470 2025-12-04T09:20:54.6505427Z * [new tag] ciflow/rocm/169471 -> ciflow/rocm/169471 2025-12-04T09:20:54.6505702Z * [new tag] ciflow/rocm/169472 -> ciflow/rocm/169472 2025-12-04T09:20:54.6506601Z * [new tag] ciflow/rocm/169514 -> ciflow/rocm/169514 2025-12-04T09:20:54.6506843Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-12-04T09:20:54.6507792Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-12-04T09:20:54.6508429Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-12-04T09:20:54.6508836Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-12-04T09:20:54.6509726Z * [new tag] ciflow/slow/167207 -> ciflow/slow/167207 2025-12-04T09:20:54.6509970Z * [new tag] ciflow/slow/168050 -> ciflow/slow/168050 2025-12-04T09:20:54.6510495Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-12-04T09:20:54.6511365Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-12-04T09:20:54.6511838Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-12-04T09:20:54.6512873Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-12-04T09:20:54.6513052Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-12-04T09:20:54.6514066Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-12-04T09:20:54.6514559Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-12-04T09:20:54.6515086Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-12-04T09:20:54.6516197Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-12-04T09:20:54.6516401Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-12-04T09:20:54.6516975Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-12-04T09:20:54.6517490Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-12-04T09:20:54.6518295Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-12-04T09:20:54.6518515Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-12-04T09:20:54.6519796Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-12-04T09:20:54.6520042Z * [new tag] ciflow/torchbench/168175 -> ciflow/torchbench/168175 2025-12-04T09:20:54.6520548Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-12-04T09:20:54.6520976Z * [new tag] ciflow/trunk/157149 -> ciflow/trunk/157149 2025-12-04T09:20:54.6521450Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-12-04T09:20:54.6521792Z * [new tag] ciflow/trunk/159718 -> ciflow/trunk/159718 2025-12-04T09:20:54.6522371Z * [new tag] ciflow/trunk/160685 -> ciflow/trunk/160685 2025-12-04T09:20:54.6522752Z * [new tag] ciflow/trunk/160729 -> ciflow/trunk/160729 2025-12-04T09:20:54.6523238Z * [new tag] ciflow/trunk/162275 -> ciflow/trunk/162275 2025-12-04T09:20:54.6524237Z * [new tag] ciflow/trunk/162795 -> ciflow/trunk/162795 2025-12-04T09:20:54.6524438Z * [new tag] ciflow/trunk/163245 -> ciflow/trunk/163245 2025-12-04T09:20:54.6524980Z * [new tag] ciflow/trunk/163942 -> ciflow/trunk/163942 2025-12-04T09:20:54.6525405Z * [new tag] ciflow/trunk/165274 -> ciflow/trunk/165274 2025-12-04T09:20:54.6526361Z * [new tag] ciflow/trunk/165483 -> ciflow/trunk/165483 2025-12-04T09:20:54.6526746Z * [new tag] ciflow/trunk/165728 -> ciflow/trunk/165728 2025-12-04T09:20:54.6527625Z * [new tag] ciflow/trunk/165922 -> ciflow/trunk/165922 2025-12-04T09:20:54.6527743Z * [new tag] ciflow/trunk/166075 -> ciflow/trunk/166075 2025-12-04T09:20:54.6528650Z * [new tag] ciflow/trunk/166165 -> ciflow/trunk/166165 2025-12-04T09:20:54.6528860Z * [new tag] ciflow/trunk/166829 -> ciflow/trunk/166829 2025-12-04T09:20:54.6529857Z * [new tag] ciflow/trunk/166843 -> ciflow/trunk/166843 2025-12-04T09:20:54.6530107Z * [new tag] ciflow/trunk/166876 -> ciflow/trunk/166876 2025-12-04T09:20:54.6530379Z * [new tag] ciflow/trunk/167207 -> ciflow/trunk/167207 2025-12-04T09:20:54.6531010Z * [new tag] ciflow/trunk/167536 -> ciflow/trunk/167536 2025-12-04T09:20:54.6531387Z * [new tag] ciflow/trunk/167552 -> ciflow/trunk/167552 2025-12-04T09:20:54.6532015Z * [new tag] ciflow/trunk/167555 -> ciflow/trunk/167555 2025-12-04T09:20:54.6532474Z * [new tag] ciflow/trunk/167599 -> ciflow/trunk/167599 2025-12-04T09:20:54.6533336Z * [new tag] ciflow/trunk/167659 -> ciflow/trunk/167659 2025-12-04T09:20:54.6533568Z * [new tag] ciflow/trunk/167672 -> ciflow/trunk/167672 2025-12-04T09:20:54.6535863Z * [new tag] ciflow/trunk/167742 -> ciflow/trunk/167742 2025-12-04T09:20:54.6536260Z * [new tag] ciflow/trunk/167781 -> ciflow/trunk/167781 2025-12-04T09:20:54.6536371Z * [new tag] ciflow/trunk/167837 -> ciflow/trunk/167837 2025-12-04T09:20:54.6536486Z * [new tag] ciflow/trunk/167887 -> ciflow/trunk/167887 2025-12-04T09:20:54.6536589Z * [new tag] ciflow/trunk/167978 -> ciflow/trunk/167978 2025-12-04T09:20:54.6536807Z * [new tag] ciflow/trunk/168050 -> ciflow/trunk/168050 2025-12-04T09:20:54.6537223Z * [new tag] ciflow/trunk/168051 -> ciflow/trunk/168051 2025-12-04T09:20:54.6537714Z * [new tag] ciflow/trunk/168096 -> ciflow/trunk/168096 2025-12-04T09:20:54.6538193Z * [new tag] ciflow/trunk/168127 -> ciflow/trunk/168127 2025-12-04T09:20:54.6538717Z * [new tag] ciflow/trunk/168157 -> ciflow/trunk/168157 2025-12-04T09:20:54.6539120Z * [new tag] ciflow/trunk/168175 -> ciflow/trunk/168175 2025-12-04T09:20:54.6539983Z * [new tag] ciflow/trunk/168209 -> ciflow/trunk/168209 2025-12-04T09:20:54.6540246Z * [new tag] ciflow/trunk/168213 -> ciflow/trunk/168213 2025-12-04T09:20:54.6540730Z * [new tag] ciflow/trunk/168226 -> ciflow/trunk/168226 2025-12-04T09:20:54.6541149Z * [new tag] ciflow/trunk/168262 -> ciflow/trunk/168262 2025-12-04T09:20:54.6541933Z * [new tag] ciflow/trunk/168275 -> ciflow/trunk/168275 2025-12-04T09:20:54.6542251Z * [new tag] ciflow/trunk/168328 -> ciflow/trunk/168328 2025-12-04T09:20:54.6543334Z * [new tag] ciflow/trunk/168368 -> ciflow/trunk/168368 2025-12-04T09:20:54.6543649Z * [new tag] ciflow/trunk/168917 -> ciflow/trunk/168917 2025-12-04T09:20:54.6543761Z * [new tag] ciflow/trunk/168933 -> ciflow/trunk/168933 2025-12-04T09:20:54.6544145Z * [new tag] ciflow/trunk/168941 -> ciflow/trunk/168941 2025-12-04T09:20:54.6544912Z * [new tag] ciflow/trunk/168955 -> ciflow/trunk/168955 2025-12-04T09:20:54.6545118Z * [new tag] ciflow/trunk/168980 -> ciflow/trunk/168980 2025-12-04T09:20:54.6548675Z * [new tag] ciflow/trunk/169004 -> ciflow/trunk/169004 2025-12-04T09:20:54.6549049Z * [new tag] ciflow/trunk/169006 -> ciflow/trunk/169006 2025-12-04T09:20:54.6549177Z * [new tag] ciflow/trunk/169023 -> ciflow/trunk/169023 2025-12-04T09:20:54.6549279Z * [new tag] ciflow/trunk/169025 -> ciflow/trunk/169025 2025-12-04T09:20:54.6549387Z * [new tag] ciflow/trunk/169048 -> ciflow/trunk/169048 2025-12-04T09:20:54.6549487Z * [new tag] ciflow/trunk/169066 -> ciflow/trunk/169066 2025-12-04T09:20:54.6549587Z * [new tag] ciflow/trunk/169091 -> ciflow/trunk/169091 2025-12-04T09:20:54.6549708Z * [new tag] ciflow/trunk/169102 -> ciflow/trunk/169102 2025-12-04T09:20:54.6549807Z * [new tag] ciflow/trunk/169103 -> ciflow/trunk/169103 2025-12-04T09:20:54.6550052Z * [new tag] ciflow/trunk/169125 -> ciflow/trunk/169125 2025-12-04T09:20:54.6550238Z * [new tag] ciflow/trunk/169139 -> ciflow/trunk/169139 2025-12-04T09:20:54.6551015Z * [new tag] ciflow/trunk/169148 -> ciflow/trunk/169148 2025-12-04T09:20:54.6551139Z * [new tag] ciflow/trunk/169151 -> ciflow/trunk/169151 2025-12-04T09:20:54.6554158Z * [new tag] ciflow/trunk/169156 -> ciflow/trunk/169156 2025-12-04T09:20:54.6554314Z * [new tag] ciflow/trunk/169176 -> ciflow/trunk/169176 2025-12-04T09:20:54.6554431Z * [new tag] ciflow/trunk/169204 -> ciflow/trunk/169204 2025-12-04T09:20:54.6554722Z * [new tag] ciflow/trunk/169207 -> ciflow/trunk/169207 2025-12-04T09:20:54.6554830Z * [new tag] ciflow/trunk/169211 -> ciflow/trunk/169211 2025-12-04T09:20:54.6554937Z * [new tag] ciflow/trunk/169231 -> ciflow/trunk/169231 2025-12-04T09:20:54.6555296Z * [new tag] ciflow/trunk/169260 -> ciflow/trunk/169260 2025-12-04T09:20:54.6555492Z * [new tag] ciflow/trunk/169271 -> ciflow/trunk/169271 2025-12-04T09:20:54.6556268Z * [new tag] ciflow/trunk/169280 -> ciflow/trunk/169280 2025-12-04T09:20:54.6556394Z * [new tag] ciflow/trunk/169281 -> ciflow/trunk/169281 2025-12-04T09:20:54.6560805Z * [new tag] ciflow/trunk/169286 -> ciflow/trunk/169286 2025-12-04T09:20:54.6560953Z * [new tag] ciflow/trunk/169293 -> ciflow/trunk/169293 2025-12-04T09:20:54.6561097Z * [new tag] ciflow/trunk/169296 -> ciflow/trunk/169296 2025-12-04T09:20:54.6561219Z * [new tag] ciflow/trunk/169304 -> ciflow/trunk/169304 2025-12-04T09:20:54.6561331Z * [new tag] ciflow/trunk/169305 -> ciflow/trunk/169305 2025-12-04T09:20:54.6561449Z * [new tag] ciflow/trunk/169312 -> ciflow/trunk/169312 2025-12-04T09:20:54.6561560Z * [new tag] ciflow/trunk/169328 -> ciflow/trunk/169328 2025-12-04T09:20:54.6561689Z * [new tag] ciflow/trunk/169343 -> ciflow/trunk/169343 2025-12-04T09:20:54.6561805Z * [new tag] ciflow/trunk/169355 -> ciflow/trunk/169355 2025-12-04T09:20:54.6561912Z * [new tag] ciflow/trunk/169370 -> ciflow/trunk/169370 2025-12-04T09:20:54.6562216Z * [new tag] ciflow/trunk/169379 -> ciflow/trunk/169379 2025-12-04T09:20:54.6562730Z * [new tag] ciflow/trunk/169380 -> ciflow/trunk/169380 2025-12-04T09:20:54.6563170Z * [new tag] ciflow/trunk/169385 -> ciflow/trunk/169385 2025-12-04T09:20:54.6563603Z * [new tag] ciflow/trunk/169387 -> ciflow/trunk/169387 2025-12-04T09:20:54.6565742Z * [new tag] ciflow/trunk/169410 -> ciflow/trunk/169410 2025-12-04T09:20:54.6566067Z * [new tag] ciflow/trunk/169412 -> ciflow/trunk/169412 2025-12-04T09:20:54.6566227Z * [new tag] ciflow/trunk/169418 -> ciflow/trunk/169418 2025-12-04T09:20:54.6566358Z * [new tag] ciflow/trunk/169423 -> ciflow/trunk/169423 2025-12-04T09:20:54.6566549Z * [new tag] ciflow/trunk/169427 -> ciflow/trunk/169427 2025-12-04T09:20:54.6566825Z * [new tag] ciflow/trunk/169430 -> ciflow/trunk/169430 2025-12-04T09:20:54.6566946Z * [new tag] ciflow/trunk/169437 -> ciflow/trunk/169437 2025-12-04T09:20:54.6567391Z * [new tag] ciflow/trunk/169442 -> ciflow/trunk/169442 2025-12-04T09:20:54.6568008Z * [new tag] ciflow/trunk/169452 -> ciflow/trunk/169452 2025-12-04T09:20:54.6568317Z * [new tag] ciflow/trunk/169454 -> ciflow/trunk/169454 2025-12-04T09:20:54.6570683Z * [new tag] ciflow/trunk/169459 -> ciflow/trunk/169459 2025-12-04T09:20:54.6570854Z * [new tag] ciflow/trunk/169474 -> ciflow/trunk/169474 2025-12-04T09:20:54.6570972Z * [new tag] ciflow/trunk/169475 -> ciflow/trunk/169475 2025-12-04T09:20:54.6571087Z * [new tag] ciflow/trunk/169476 -> ciflow/trunk/169476 2025-12-04T09:20:54.6571254Z * [new tag] ciflow/trunk/169487 -> ciflow/trunk/169487 2025-12-04T09:20:54.6571807Z * [new tag] ciflow/trunk/169497 -> ciflow/trunk/169497 2025-12-04T09:20:54.6572451Z * [new tag] ciflow/trunk/169503 -> ciflow/trunk/169503 2025-12-04T09:20:54.6572887Z * [new tag] ciflow/trunk/169505 -> ciflow/trunk/169505 2025-12-04T09:20:54.6575293Z * [new tag] ciflow/trunk/169507 -> ciflow/trunk/169507 2025-12-04T09:20:54.6575604Z * [new tag] ciflow/trunk/169514 -> ciflow/trunk/169514 2025-12-04T09:20:54.6575932Z * [new tag] ciflow/trunk/169517 -> ciflow/trunk/169517 2025-12-04T09:20:54.6576188Z * [new tag] ciflow/trunk/169519 -> ciflow/trunk/169519 2025-12-04T09:20:54.6576321Z * [new tag] ciflow/trunk/169528 -> ciflow/trunk/169528 2025-12-04T09:20:54.6576515Z * [new tag] ciflow/trunk/169541 -> ciflow/trunk/169541 2025-12-04T09:20:54.6577188Z * [new tag] ciflow/trunk/169555 -> ciflow/trunk/169555 2025-12-04T09:20:54.6577487Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-12-04T09:20:54.6577878Z * [new tag] ciflow/vllm/165270 -> ciflow/vllm/165270 2025-12-04T09:20:54.6578903Z * [new tag] ciflow/vllm/165274 -> ciflow/vllm/165274 2025-12-04T09:20:54.6579050Z * [new tag] ciflow/vllm/166494 -> ciflow/vllm/166494 2025-12-04T09:20:54.6579282Z * [new tag] ciflow/vllm/169219 -> ciflow/vllm/169219 2025-12-04T09:20:54.6581241Z * [new tag] ciflow/vllm/169220 -> ciflow/vllm/169220 2025-12-04T09:20:54.6581540Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-12-04T09:20:54.6581674Z * [new tag] ciflow/xpu/159718 -> ciflow/xpu/159718 2025-12-04T09:20:54.6581787Z * [new tag] ciflow/xpu/161940 -> ciflow/xpu/161940 2025-12-04T09:20:54.6582022Z * [new tag] ciflow/xpu/163251 -> ciflow/xpu/163251 2025-12-04T09:20:54.6582171Z * [new tag] ciflow/xpu/166829 -> ciflow/xpu/166829 2025-12-04T09:20:54.6584401Z * [new tag] ciflow/xpu/166843 -> ciflow/xpu/166843 2025-12-04T09:20:54.6584700Z * [new tag] ciflow/xpu/167972 -> ciflow/xpu/167972 2025-12-04T09:20:54.6584831Z * [new tag] ciflow/xpu/167981 -> ciflow/xpu/167981 2025-12-04T09:20:54.6584949Z * [new tag] ciflow/xpu/168213 -> ciflow/xpu/168213 2025-12-04T09:20:54.6585188Z * [new tag] ciflow/xpu/168262 -> ciflow/xpu/168262 2025-12-04T09:20:54.6585315Z * [new tag] ciflow/xpu/168328 -> ciflow/xpu/168328 2025-12-04T09:20:54.6585682Z * [new tag] ciflow/xpu/168950 -> ciflow/xpu/168950 2025-12-04T09:20:54.6591169Z * [new tag] ciflow/xpu/169039 -> ciflow/xpu/169039 2025-12-04T09:20:54.6591468Z * [new tag] ciflow/xpu/169200 -> ciflow/xpu/169200 2025-12-04T09:20:54.6591644Z * [new tag] ciflow/xpu/169203 -> ciflow/xpu/169203 2025-12-04T09:20:54.6591780Z * [new tag] ciflow/xpu/169230 -> ciflow/xpu/169230 2025-12-04T09:20:54.6591886Z * [new tag] ciflow/xpu/169231 -> ciflow/xpu/169231 2025-12-04T09:20:54.6592002Z * [new tag] ciflow/xpu/169241 -> ciflow/xpu/169241 2025-12-04T09:20:54.6592107Z * [new tag] ciflow/xpu/169280 -> ciflow/xpu/169280 2025-12-04T09:20:54.6592216Z * [new tag] ciflow/xpu/169296 -> ciflow/xpu/169296 2025-12-04T09:20:54.6592320Z * [new tag] ciflow/xpu/169353 -> ciflow/xpu/169353 2025-12-04T09:20:54.6592428Z * [new tag] ciflow/xpu/169410 -> ciflow/xpu/169410 2025-12-04T09:20:54.6592548Z * [new tag] ciflow/xpu/169442 -> ciflow/xpu/169442 2025-12-04T09:20:54.6592736Z * [new tag] ciflow/xpu/169555 -> ciflow/xpu/169555 2025-12-04T09:20:54.6592999Z * [new tag] cslpull75 -> cslpull75 2025-12-04T09:20:54.6593100Z * [new tag] cslpull76 -> cslpull76 2025-12-04T09:20:54.6593522Z * [new tag] cslpull77 -> cslpull77 2025-12-04T09:20:54.6596254Z * [new tag] cslpull78 -> cslpull78 2025-12-04T09:20:54.6596599Z * [new tag] cslpull79 -> cslpull79 2025-12-04T09:20:54.6596778Z * [new tag] cslpull80 -> cslpull80 2025-12-04T09:20:54.6596892Z * [new tag] cslpull81 -> cslpull81 2025-12-04T09:20:54.6597104Z * [new tag] cslpull82 -> cslpull82 2025-12-04T09:20:54.6597366Z * [new tag] cslpull83 -> cslpull83 2025-12-04T09:20:54.6597486Z * [new tag] cslpull84 -> cslpull84 2025-12-04T09:20:54.6600144Z * [new tag] cslpull85 -> cslpull85 2025-12-04T09:20:54.6600413Z * [new tag] cslpull86 -> cslpull86 2025-12-04T09:20:54.6600554Z * [new tag] cslpull87 -> cslpull87 2025-12-04T09:20:54.6600664Z * [new tag] cslpull88 -> cslpull88 2025-12-04T09:20:54.6600812Z * [new tag] cslpull89 -> cslpull89 2025-12-04T09:20:54.6600915Z * [new tag] cslpull90 -> cslpull90 2025-12-04T09:20:54.6602138Z * [new tag] cslpull91 -> cslpull91 2025-12-04T09:20:54.6602437Z * [new tag] cslpull92 -> cslpull92 2025-12-04T09:20:54.6602620Z * [new tag] flight_5 -> flight_5 2025-12-04T09:20:54.6605109Z * [new tag] flight_5.1 -> flight_5.1 2025-12-04T09:20:54.6605442Z * [new tag] flight_5.2 -> flight_5.2 2025-12-04T09:20:54.6605705Z * [new tag] flight_5.3 -> flight_5.3 2025-12-04T09:20:54.6605884Z * [new tag] forpull1 -> forpull1 2025-12-04T09:20:54.6607410Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-12-04T09:20:54.6607583Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-12-04T09:20:54.6607716Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-12-04T09:20:54.6608016Z * [new tag] nightly-binary -> nightly-binary 2025-12-04T09:20:54.6610922Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-12-04T09:20:54.6611093Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-12-04T09:20:54.6611378Z * [new tag] trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 -> trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 2025-12-04T09:20:54.6611917Z * [new tag] trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e -> trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e 2025-12-04T09:20:54.6612193Z * [new tag] trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 -> trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 2025-12-04T09:20:54.6617168Z * [new tag] trunk/07dcc0b83db3211653a38565a24e15acdba75654 -> trunk/07dcc0b83db3211653a38565a24e15acdba75654 2025-12-04T09:20:54.6619380Z * [new tag] trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb -> trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb 2025-12-04T09:20:54.6619766Z * [new tag] trunk/088048f2fea28ff7d450f65c72419ca45780d30b -> trunk/088048f2fea28ff7d450f65c72419ca45780d30b 2025-12-04T09:20:54.6625638Z * [new tag] trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 -> trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 2025-12-04T09:20:54.6630485Z * [new tag] trunk/0b80a4c62b94402844bf221791c096b0035c6d75 -> trunk/0b80a4c62b94402844bf221791c096b0035c6d75 2025-12-04T09:20:54.6632776Z * [new tag] trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 -> trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 2025-12-04T09:20:54.6635821Z * [new tag] trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 -> trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 2025-12-04T09:20:54.6636146Z * [new tag] trunk/135f3753c418a6879b1954904184937b67e61688 -> trunk/135f3753c418a6879b1954904184937b67e61688 2025-12-04T09:20:54.6641908Z * [new tag] trunk/15da21026cb13cd20257dc9e96830db108743c10 -> trunk/15da21026cb13cd20257dc9e96830db108743c10 2025-12-04T09:20:54.6647021Z * [new tag] trunk/166efdad2ac827f30fb02504c6017520257f88ec -> trunk/166efdad2ac827f30fb02504c6017520257f88ec 2025-12-04T09:20:54.6647329Z * [new tag] trunk/174272c15fae553d8488140af931f7d8050a313f -> trunk/174272c15fae553d8488140af931f7d8050a313f 2025-12-04T09:20:54.6647619Z * [new tag] trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 -> trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 2025-12-04T09:20:54.6647894Z * [new tag] trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 -> trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 2025-12-04T09:20:54.6648147Z * [new tag] trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 -> trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 2025-12-04T09:20:54.6648404Z * [new tag] trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 -> trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 2025-12-04T09:20:54.6648660Z * [new tag] trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e -> trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e 2025-12-04T09:20:54.6648909Z * [new tag] trunk/1c87554d74140eaee964ca8b1832cede67f5f520 -> trunk/1c87554d74140eaee964ca8b1832cede67f5f520 2025-12-04T09:20:54.6649157Z * [new tag] trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 -> trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 2025-12-04T09:20:54.6649411Z * [new tag] trunk/1cee47d6ce0a02227185b566593f002dd639ca0c -> trunk/1cee47d6ce0a02227185b566593f002dd639ca0c 2025-12-04T09:20:54.6649670Z * [new tag] trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d -> trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d 2025-12-04T09:20:54.6649927Z * [new tag] trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 -> trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 2025-12-04T09:20:54.6650194Z * [new tag] trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de -> trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de 2025-12-04T09:20:54.6650450Z * [new tag] trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 -> trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 2025-12-04T09:20:54.6650698Z * [new tag] trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 -> trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 2025-12-04T09:20:54.6650953Z * [new tag] trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f -> trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f 2025-12-04T09:20:54.6651191Z * [new tag] trunk/285779b1621cf9f073a062b0889a642d200308d9 -> trunk/285779b1621cf9f073a062b0889a642d200308d9 2025-12-04T09:20:54.6651622Z * [new tag] trunk/2887faaec6295d081580d09fce161201826c6d87 -> trunk/2887faaec6295d081580d09fce161201826c6d87 2025-12-04T09:20:54.6651892Z * [new tag] trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc -> trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc 2025-12-04T09:20:54.6652141Z * [new tag] trunk/29856679769b3dede478767e2fe6cfb51197cb25 -> trunk/29856679769b3dede478767e2fe6cfb51197cb25 2025-12-04T09:20:54.6652398Z * [new tag] trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 -> trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 2025-12-04T09:20:54.6652651Z * [new tag] trunk/2ac3ef882afb23136adc188975f0a8802fc68adf -> trunk/2ac3ef882afb23136adc188975f0a8802fc68adf 2025-12-04T09:20:54.6653120Z * [new tag] trunk/2bec68e73b64715354af076ad309335f943e36cd -> trunk/2bec68e73b64715354af076ad309335f943e36cd 2025-12-04T09:20:54.6653342Z * [new tag] trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 -> trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 2025-12-04T09:20:54.6653608Z * [new tag] trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 -> trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 2025-12-04T09:20:54.6653912Z * [new tag] trunk/2df6058f116a65722a0e03073402feb242572d35 -> trunk/2df6058f116a65722a0e03073402feb242572d35 2025-12-04T09:20:54.6654154Z * [new tag] trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec -> trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec 2025-12-04T09:20:54.6654455Z * [new tag] trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 -> trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 2025-12-04T09:20:54.6654697Z * [new tag] trunk/305168768a95d69c444df5cd334bb774edfe06f1 -> trunk/305168768a95d69c444df5cd334bb774edfe06f1 2025-12-04T09:20:54.6654939Z * [new tag] trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 -> trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 2025-12-04T09:20:54.6655189Z * [new tag] trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 -> trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 2025-12-04T09:20:54.6655406Z * [new tag] trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 -> trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 2025-12-04T09:20:54.6655634Z * [new tag] trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf -> trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf 2025-12-04T09:20:54.6655856Z * [new tag] trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee -> trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee 2025-12-04T09:20:54.6656086Z * [new tag] trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 -> trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 2025-12-04T09:20:54.6656313Z * [new tag] trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 -> trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 2025-12-04T09:20:54.6656544Z * [new tag] trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae -> trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae 2025-12-04T09:20:54.6656781Z * [new tag] trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f -> trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f 2025-12-04T09:20:54.6657007Z * [new tag] trunk/42e9005cda22da3f1c559c3649218cebd671027c -> trunk/42e9005cda22da3f1c559c3649218cebd671027c 2025-12-04T09:20:54.6657234Z * [new tag] trunk/43b94713bbf340d3c124fde02d0f73add4021247 -> trunk/43b94713bbf340d3c124fde02d0f73add4021247 2025-12-04T09:20:54.6657457Z * [new tag] trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c -> trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c 2025-12-04T09:20:54.6657708Z * [new tag] trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a -> trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a 2025-12-04T09:20:54.6657940Z * [new tag] trunk/45d310ad84854dff730c0b12e577d7998d978686 -> trunk/45d310ad84854dff730c0b12e577d7998d978686 2025-12-04T09:20:54.6658188Z * [new tag] trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 -> trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 2025-12-04T09:20:54.6658430Z * [new tag] trunk/481e5ab336275bd3acd5fa8a611b05b4469012af -> trunk/481e5ab336275bd3acd5fa8a611b05b4469012af 2025-12-04T09:20:54.6658659Z * [new tag] trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 -> trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 2025-12-04T09:20:54.6658906Z * [new tag] trunk/49a04d26088acc17d948ddd66920f3e16371e873 -> trunk/49a04d26088acc17d948ddd66920f3e16371e873 2025-12-04T09:20:54.6659156Z * [new tag] trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 -> trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 2025-12-04T09:20:54.6659452Z * [new tag] trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f -> trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f 2025-12-04T09:20:54.6659698Z * [new tag] trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa -> trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa 2025-12-04T09:20:54.6659972Z * [new tag] trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c -> trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c 2025-12-04T09:20:54.6660213Z * [new tag] trunk/4fefb8e7e942386ffac764a41b232241f82bea3a -> trunk/4fefb8e7e942386ffac764a41b232241f82bea3a 2025-12-04T09:20:54.6660434Z * [new tag] trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d -> trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d 2025-12-04T09:20:54.6660720Z * [new tag] trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 -> trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 2025-12-04T09:20:54.6660943Z * [new tag] trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 -> trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 2025-12-04T09:20:54.6661171Z * [new tag] trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a -> trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a 2025-12-04T09:20:54.6661412Z * [new tag] trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 -> trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 2025-12-04T09:20:54.6661671Z * [new tag] trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 -> trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 2025-12-04T09:20:54.6661893Z * [new tag] trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 -> trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 2025-12-04T09:20:54.6662138Z * [new tag] trunk/5634469fda9e5d98869c82c7d03bb08914245f96 -> trunk/5634469fda9e5d98869c82c7d03bb08914245f96 2025-12-04T09:20:54.6662374Z * [new tag] trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc -> trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc 2025-12-04T09:20:54.6662627Z * [new tag] trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 -> trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 2025-12-04T09:20:54.6662858Z * [new tag] trunk/597930f6b568852356ca9795dac76f9e4653adbd -> trunk/597930f6b568852356ca9795dac76f9e4653adbd 2025-12-04T09:20:54.6663113Z * [new tag] trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 -> trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 2025-12-04T09:20:54.6663359Z * [new tag] trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 -> trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 2025-12-04T09:20:54.6663590Z * [new tag] trunk/5a607febc04c3a2b5824c75f3f60307867439a2c -> trunk/5a607febc04c3a2b5824c75f3f60307867439a2c 2025-12-04T09:20:54.6663842Z * [new tag] trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b -> trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b 2025-12-04T09:20:54.6664078Z * [new tag] trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c -> trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c 2025-12-04T09:20:54.6664313Z * [new tag] trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 -> trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 2025-12-04T09:20:54.6664556Z * [new tag] trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 -> trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 2025-12-04T09:20:54.6664803Z * [new tag] trunk/61be54a31dc09b59d99b62176fb935aee0b924ef -> trunk/61be54a31dc09b59d99b62176fb935aee0b924ef 2025-12-04T09:20:54.6665039Z * [new tag] trunk/62d3ccd71484ed6a760d909b41487101bbc65719 -> trunk/62d3ccd71484ed6a760d909b41487101bbc65719 2025-12-04T09:20:54.6665279Z * [new tag] trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b -> trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b 2025-12-04T09:20:54.6665518Z * [new tag] trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a -> trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a 2025-12-04T09:20:54.6665739Z * [new tag] trunk/66004b993744b4106bf8afaba71f3c228a804206 -> trunk/66004b993744b4106bf8afaba71f3c228a804206 2025-12-04T09:20:54.6666045Z * [new tag] trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 -> trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 2025-12-04T09:20:54.6666285Z * [new tag] trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 -> trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 2025-12-04T09:20:54.6666558Z * [new tag] trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d -> trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d 2025-12-04T09:20:54.6666801Z * [new tag] trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b -> trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b 2025-12-04T09:20:54.6667031Z * [new tag] trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 -> trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 2025-12-04T09:20:54.6667255Z * [new tag] trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 -> trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 2025-12-04T09:20:54.6667490Z * [new tag] trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec -> trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec 2025-12-04T09:20:54.6667712Z * [new tag] trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 -> trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 2025-12-04T09:20:54.6667958Z * [new tag] trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d -> trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d 2025-12-04T09:20:54.6668566Z * [new tag] trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a -> trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a 2025-12-04T09:20:54.6669175Z * [new tag] trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e -> trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e 2025-12-04T09:20:54.6669737Z * [new tag] trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 -> trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 2025-12-04T09:20:54.6670387Z * [new tag] trunk/70d797a5fc109b20a517646fcaa819477cd0d485 -> trunk/70d797a5fc109b20a517646fcaa819477cd0d485 2025-12-04T09:20:54.6670914Z * [new tag] trunk/7348cb355ff0a6f79cd4871215aea72185748734 -> trunk/7348cb355ff0a6f79cd4871215aea72185748734 2025-12-04T09:20:54.6671984Z * [new tag] trunk/74fe26a1ebe32931783569f2e762e3c2c974901f -> trunk/74fe26a1ebe32931783569f2e762e3c2c974901f 2025-12-04T09:20:54.6672770Z * [new tag] trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 -> trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 2025-12-04T09:20:54.6676852Z * [new tag] trunk/7716da9fb23f27a65b41f9f016a2afadf281c18f -> trunk/7716da9fb23f27a65b41f9f016a2afadf281c18f 2025-12-04T09:20:54.6677240Z * [new tag] trunk/7741edd4ed665f3988052e260863efb508d61a03 -> trunk/7741edd4ed665f3988052e260863efb508d61a03 2025-12-04T09:20:54.6678948Z * [new tag] trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 -> trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 2025-12-04T09:20:54.6679220Z * [new tag] trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 -> trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 2025-12-04T09:20:54.6679482Z * [new tag] trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 -> trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 2025-12-04T09:20:54.6679736Z * [new tag] trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca -> trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca 2025-12-04T09:20:54.6679983Z * [new tag] trunk/7b7af390ea8541c611d1ce2018a6934188fc197b -> trunk/7b7af390ea8541c611d1ce2018a6934188fc197b 2025-12-04T09:20:54.6680990Z * [new tag] trunk/7ba4680f3755a560af81aa0f688791e367aa3609 -> trunk/7ba4680f3755a560af81aa0f688791e367aa3609 2025-12-04T09:20:54.6681246Z * [new tag] trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b -> trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b 2025-12-04T09:20:54.6682307Z * [new tag] trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T09:20:54.6682802Z * [new tag] trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 -> trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 2025-12-04T09:20:54.6683083Z * [new tag] trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed -> trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed 2025-12-04T09:20:54.6683778Z * [new tag] trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 -> trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 2025-12-04T09:20:54.6684339Z * [new tag] trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e -> trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e 2025-12-04T09:20:54.6684698Z * [new tag] trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead -> trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead 2025-12-04T09:20:54.6685699Z * [new tag] trunk/81af382128efa094d8702e18f2c133760904c718 -> trunk/81af382128efa094d8702e18f2c133760904c718 2025-12-04T09:20:54.6686158Z * [new tag] trunk/84149583d483e9c973c9a0feda70e4f3964947b0 -> trunk/84149583d483e9c973c9a0feda70e4f3964947b0 2025-12-04T09:20:54.6687380Z * [new tag] trunk/85a315917efe82c24306be805c584ec044951c75 -> trunk/85a315917efe82c24306be805c584ec044951c75 2025-12-04T09:20:54.6688174Z * [new tag] trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece -> trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece 2025-12-04T09:20:54.6688473Z * [new tag] trunk/892640e25aeefa8007c5af837214b4502b6b62a6 -> trunk/892640e25aeefa8007c5af837214b4502b6b62a6 2025-12-04T09:20:54.6689027Z * [new tag] trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 -> trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 2025-12-04T09:20:54.6689569Z * [new tag] trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c -> trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c 2025-12-04T09:20:54.6690255Z * [new tag] trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 -> trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 2025-12-04T09:20:54.6690914Z * [new tag] trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 -> trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 2025-12-04T09:20:54.6691859Z * [new tag] trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca -> trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca 2025-12-04T09:20:54.6692194Z * [new tag] trunk/90b27e7e8352cde97d32ddad24740ef819633f38 -> trunk/90b27e7e8352cde97d32ddad24740ef819633f38 2025-12-04T09:20:54.6692764Z * [new tag] trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 -> trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 2025-12-04T09:20:54.6693310Z * [new tag] trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c -> trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c 2025-12-04T09:20:54.6693995Z * [new tag] trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 -> trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 2025-12-04T09:20:54.6694453Z * [new tag] trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 -> trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 2025-12-04T09:20:54.6695403Z * [new tag] trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa -> trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa 2025-12-04T09:20:54.6696034Z * [new tag] trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d -> trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d 2025-12-04T09:20:54.6696368Z * [new tag] trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 -> trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 2025-12-04T09:20:54.6697089Z * [new tag] trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 -> trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 2025-12-04T09:20:54.6697679Z * [new tag] trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d -> trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d 2025-12-04T09:20:54.6698400Z * [new tag] trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a -> trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a 2025-12-04T09:20:54.6698929Z * [new tag] trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 -> trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 2025-12-04T09:20:54.6699653Z * [new tag] trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 -> trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 2025-12-04T09:20:54.6700299Z * [new tag] trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa -> trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa 2025-12-04T09:20:54.6701273Z * [new tag] trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d -> trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d 2025-12-04T09:20:54.6701655Z * [new tag] trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c -> trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c 2025-12-04T09:20:54.6702265Z * [new tag] trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 -> trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 2025-12-04T09:20:54.6702887Z * [new tag] trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c -> trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c 2025-12-04T09:20:54.6703315Z * [new tag] trunk/a7dc6dab9ad911259d4801c502907e531594db45 -> trunk/a7dc6dab9ad911259d4801c502907e531594db45 2025-12-04T09:20:54.6704016Z * [new tag] trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 -> trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 2025-12-04T09:20:54.6704637Z * [new tag] trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e -> trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e 2025-12-04T09:20:54.6705214Z * [new tag] trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e -> trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e 2025-12-04T09:20:54.6705894Z * [new tag] trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e -> trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e 2025-12-04T09:20:54.6706262Z * [new tag] trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 -> trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 2025-12-04T09:20:54.6706850Z * [new tag] trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 -> trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 2025-12-04T09:20:54.6707492Z * [new tag] trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 -> trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 2025-12-04T09:20:54.6708095Z * [new tag] trunk/b39813b4a04931682b0491adba2138d01d716d99 -> trunk/b39813b4a04931682b0491adba2138d01d716d99 2025-12-04T09:20:54.6708888Z * [new tag] trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 -> trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 2025-12-04T09:20:54.6709292Z * [new tag] trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 -> trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 2025-12-04T09:20:54.6709952Z * [new tag] trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a -> trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a 2025-12-04T09:20:54.6710594Z * [new tag] trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 -> trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 2025-12-04T09:20:54.6711240Z * [new tag] trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 -> trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 2025-12-04T09:20:54.6711831Z * [new tag] trunk/b7d60685f8cbc939b68a20871e90db67e729329b -> trunk/b7d60685f8cbc939b68a20871e90db67e729329b 2025-12-04T09:20:54.6712444Z * [new tag] trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e -> trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e 2025-12-04T09:20:54.6713076Z * [new tag] trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf -> trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf 2025-12-04T09:20:54.6713645Z * [new tag] trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 -> trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 2025-12-04T09:20:54.6714486Z * [new tag] trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f -> trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f 2025-12-04T09:20:54.6714805Z * [new tag] trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f -> trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f 2025-12-04T09:20:54.6716117Z * [new tag] trunk/bb3034198b459401fabeab254e1b99f0115046e2 -> trunk/bb3034198b459401fabeab254e1b99f0115046e2 2025-12-04T09:20:54.6716348Z * [new tag] trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 -> trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 2025-12-04T09:20:54.6719173Z * [new tag] trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 -> trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 2025-12-04T09:20:54.6719460Z * [new tag] trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 -> trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 2025-12-04T09:20:54.6719726Z * [new tag] trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 -> trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 2025-12-04T09:20:54.6719966Z * [new tag] trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 -> trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 2025-12-04T09:20:54.6720212Z * [new tag] trunk/c0660bcee27e7d7731634e274576a7081882bede -> trunk/c0660bcee27e7d7731634e274576a7081882bede 2025-12-04T09:20:54.6720470Z * [new tag] trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac -> trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac 2025-12-04T09:20:54.6721172Z * [new tag] trunk/c55b1e8f61d041ee436d697449eb028931d574fb -> trunk/c55b1e8f61d041ee436d697449eb028931d574fb 2025-12-04T09:20:54.6721413Z * [new tag] trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 -> trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 2025-12-04T09:20:54.6725191Z * [new tag] trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 -> trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 2025-12-04T09:20:54.6725464Z * [new tag] trunk/cc0853af42122f8185321f542616f4474e717f09 -> trunk/cc0853af42122f8185321f542616f4474e717f09 2025-12-04T09:20:54.6725705Z * [new tag] trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 -> trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 2025-12-04T09:20:54.6725963Z * [new tag] trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a -> trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a 2025-12-04T09:20:54.6726199Z * [new tag] trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace -> trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace 2025-12-04T09:20:54.6726437Z * [new tag] trunk/d16447dacaf2420ea175f0c275c75da951f57d39 -> trunk/d16447dacaf2420ea175f0c275c75da951f57d39 2025-12-04T09:20:54.6726675Z * [new tag] trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 -> trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 2025-12-04T09:20:54.6726907Z * [new tag] trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 -> trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 2025-12-04T09:20:54.6727181Z * [new tag] trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf -> trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf 2025-12-04T09:20:54.6727732Z * [new tag] trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 -> trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 2025-12-04T09:20:54.6728217Z * [new tag] trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d -> trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d 2025-12-04T09:20:54.6728872Z * [new tag] trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 -> trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 2025-12-04T09:20:54.6729499Z * [new tag] trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 -> trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 2025-12-04T09:20:54.6730077Z * [new tag] trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e -> trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e 2025-12-04T09:20:54.6730699Z * [new tag] trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a -> trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a 2025-12-04T09:20:54.6731318Z * [new tag] trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b -> trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b 2025-12-04T09:20:54.6731906Z * [new tag] trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec -> trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec 2025-12-04T09:20:54.6733004Z * [new tag] trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf -> trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf 2025-12-04T09:20:54.6733432Z * [new tag] trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd -> trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd 2025-12-04T09:20:54.6733943Z * [new tag] trunk/dd18a75336a4fbd7497955cc5665904724fce889 -> trunk/dd18a75336a4fbd7497955cc5665904724fce889 2025-12-04T09:20:54.6734657Z * [new tag] trunk/ded9bcd61a059bf723e6e84689552962b480ea77 -> trunk/ded9bcd61a059bf723e6e84689552962b480ea77 2025-12-04T09:20:54.6735778Z * [new tag] trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c -> trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c 2025-12-04T09:20:54.6736430Z * [new tag] trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b -> trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b 2025-12-04T09:20:54.6736791Z * [new tag] trunk/e3f24fd73ad74c6e7176687986436956c7c18235 -> trunk/e3f24fd73ad74c6e7176687986436956c7c18235 2025-12-04T09:20:54.6737226Z * [new tag] trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e -> trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e 2025-12-04T09:20:54.6738712Z * [new tag] trunk/ea7035f462a0d2830865ee86c832bd101e1427fc -> trunk/ea7035f462a0d2830865ee86c832bd101e1427fc 2025-12-04T09:20:54.6738987Z * [new tag] trunk/eabb7ad2128580ef674446027b95bcf4e21e8df3 -> trunk/eabb7ad2128580ef674446027b95bcf4e21e8df3 2025-12-04T09:20:54.6739352Z * [new tag] trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf -> trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf 2025-12-04T09:20:54.6739966Z * [new tag] trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e -> trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e 2025-12-04T09:20:54.6740580Z * [new tag] trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e -> trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e 2025-12-04T09:20:54.6746052Z * [new tag] trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 -> trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 2025-12-04T09:20:54.6746329Z * [new tag] trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 -> trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 2025-12-04T09:20:54.6746594Z * [new tag] trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 -> trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 2025-12-04T09:20:54.6746826Z * [new tag] trunk/f1076f5510920044912247b1abb8760cb820f598 -> trunk/f1076f5510920044912247b1abb8760cb820f598 2025-12-04T09:20:54.6747057Z * [new tag] trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 -> trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 2025-12-04T09:20:54.6747293Z * [new tag] trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 -> trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 2025-12-04T09:20:54.6747532Z * [new tag] trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 -> trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 2025-12-04T09:20:54.6747761Z * [new tag] trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 -> trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 2025-12-04T09:20:54.6747993Z * [new tag] trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 -> trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 2025-12-04T09:20:54.6748217Z * [new tag] trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 -> trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 2025-12-04T09:20:54.6748441Z * [new tag] trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 -> trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 2025-12-04T09:20:54.6748664Z * [new tag] trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b -> trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b 2025-12-04T09:20:54.6748887Z * [new tag] trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 -> trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 2025-12-04T09:20:54.6749456Z * [new tag] trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 -> trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 2025-12-04T09:20:54.6749706Z * [new tag] trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 -> trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 2025-12-04T09:20:54.6750282Z * [new tag] trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 -> trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:20:54.6751558Z * [new tag] v0.1.1 -> v0.1.1 2025-12-04T09:20:54.6751852Z * [new tag] v0.1.10 -> v0.1.10 2025-12-04T09:20:54.6752008Z * [new tag] v0.1.11 -> v0.1.11 2025-12-04T09:20:54.6755106Z * [new tag] v0.1.12 -> v0.1.12 2025-12-04T09:20:54.6755373Z * [new tag] v0.1.2 -> v0.1.2 2025-12-04T09:20:54.6755689Z * [new tag] v0.1.3 -> v0.1.3 2025-12-04T09:20:54.6756001Z * [new tag] v0.1.4 -> v0.1.4 2025-12-04T09:20:54.6756112Z * [new tag] v0.1.5 -> v0.1.5 2025-12-04T09:20:54.6756198Z * [new tag] v0.1.6 -> v0.1.6 2025-12-04T09:20:54.6756292Z * [new tag] v0.1.7 -> v0.1.7 2025-12-04T09:20:54.6756498Z * [new tag] v0.1.8 -> v0.1.8 2025-12-04T09:20:54.6757554Z * [new tag] v0.1.9 -> v0.1.9 2025-12-04T09:20:54.6757684Z * [new tag] v0.2.0 -> v0.2.0 2025-12-04T09:20:54.6758113Z * [new tag] v0.3.0 -> v0.3.0 2025-12-04T09:20:54.6761085Z * [new tag] v0.3.1 -> v0.3.1 2025-12-04T09:20:54.6761338Z * [new tag] v0.4.0 -> v0.4.0 2025-12-04T09:20:54.6761511Z * [new tag] v0.4.1 -> v0.4.1 2025-12-04T09:20:54.6761626Z * [new tag] v1.0.0 -> v1.0.0 2025-12-04T09:20:54.6761907Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-12-04T09:20:54.6762021Z * [new tag] v1.0.1 -> v1.0.1 2025-12-04T09:20:54.6762127Z * [new tag] v1.0rc0 -> v1.0rc0 2025-12-04T09:20:54.6763598Z * [new tag] v1.0rc1 -> v1.0rc1 2025-12-04T09:20:54.6763929Z * [new tag] v1.1.0 -> v1.1.0 2025-12-04T09:20:54.6764027Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-12-04T09:20:54.6765944Z * [new tag] v1.10.0 -> v1.10.0 2025-12-04T09:20:54.6766087Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-12-04T09:20:54.6766195Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-12-04T09:20:54.6766305Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-12-04T09:20:54.6766413Z * [new tag] v1.10.1 -> v1.10.1 2025-12-04T09:20:54.6766695Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-12-04T09:20:54.6767936Z * [new tag] v1.10.2 -> v1.10.2 2025-12-04T09:20:54.6768076Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-12-04T09:20:54.6768959Z * [new tag] v1.11.0 -> v1.11.0 2025-12-04T09:20:54.6769386Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-12-04T09:20:54.6770183Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-12-04T09:20:54.6770679Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-12-04T09:20:54.6771831Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-12-04T09:20:54.6772100Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-12-04T09:20:54.6772513Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-12-04T09:20:54.6772942Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-12-04T09:20:54.6775258Z * [new tag] v1.12.0 -> v1.12.0 2025-12-04T09:20:54.6775609Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-12-04T09:20:54.6775739Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-12-04T09:20:54.6775870Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-12-04T09:20:54.6775989Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-12-04T09:20:54.6776762Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-12-04T09:20:54.6777196Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-12-04T09:20:54.6779684Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-12-04T09:20:54.6780000Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-12-04T09:20:54.6780269Z * [new tag] v1.12.1 -> v1.12.1 2025-12-04T09:20:54.6780569Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-12-04T09:20:54.6780687Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-12-04T09:20:54.6780790Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-12-04T09:20:54.6781021Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-12-04T09:20:54.6783299Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-12-04T09:20:54.6783471Z * [new tag] v1.13.0 -> v1.13.0 2025-12-04T09:20:54.6783592Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-12-04T09:20:54.6783704Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-12-04T09:20:54.6783855Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-12-04T09:20:54.6784622Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-12-04T09:20:54.6784835Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-12-04T09:20:54.6785470Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-12-04T09:20:54.6786327Z * [new tag] v1.13.1 -> v1.13.1 2025-12-04T09:20:54.6786487Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-12-04T09:20:54.6787314Z * [new tag] v1.2.0 -> v1.2.0 2025-12-04T09:20:54.6787440Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-12-04T09:20:54.6791284Z * [new tag] v1.3.0 -> v1.3.0 2025-12-04T09:20:54.6791586Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-12-04T09:20:54.6791863Z * [new tag] v1.3.1 -> v1.3.1 2025-12-04T09:20:54.6791991Z * [new tag] v1.4.0 -> v1.4.0 2025-12-04T09:20:54.6792099Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-12-04T09:20:54.6792202Z * [new tag] v1.4.1 -> v1.4.1 2025-12-04T09:20:54.6792308Z * [new tag] v1.5.0 -> v1.5.0 2025-12-04T09:20:54.6792420Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-12-04T09:20:54.6792553Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-12-04T09:20:54.6794351Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-12-04T09:20:54.6794644Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-12-04T09:20:54.6794790Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-12-04T09:20:54.6795252Z * [new tag] v1.5.1 -> v1.5.1 2025-12-04T09:20:54.6795359Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-12-04T09:20:54.6795458Z * [new tag] v1.6.0 -> v1.6.0 2025-12-04T09:20:54.6797348Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-12-04T09:20:54.6797653Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-12-04T09:20:54.6797760Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-12-04T09:20:54.6797911Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-12-04T09:20:54.6799583Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-12-04T09:20:54.6799856Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-12-04T09:20:54.6800017Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-12-04T09:20:54.6800152Z * [new tag] v1.7.0 -> v1.7.0 2025-12-04T09:20:54.6800540Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-12-04T09:20:54.6801995Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-12-04T09:20:54.6802152Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-12-04T09:20:54.6802285Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-12-04T09:20:54.6804884Z * [new tag] v1.7.1 -> v1.7.1 2025-12-04T09:20:54.6805153Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-12-04T09:20:54.6805316Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-12-04T09:20:54.6805421Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-12-04T09:20:54.6805688Z * [new tag] v1.8.0 -> v1.8.0 2025-12-04T09:20:54.6805802Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-12-04T09:20:54.6806794Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-12-04T09:20:54.6807485Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-12-04T09:20:54.6807761Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-12-04T09:20:54.6807990Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-12-04T09:20:54.6808752Z * [new tag] v1.8.1 -> v1.8.1 2025-12-04T09:20:54.6808980Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-12-04T09:20:54.6809585Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-12-04T09:20:54.6809867Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-12-04T09:20:54.6811223Z * [new tag] v1.8.2 -> v1.8.2 2025-12-04T09:20:54.6811579Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-12-04T09:20:54.6812079Z * [new tag] v1.9.0 -> v1.9.0 2025-12-04T09:20:54.6816510Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-12-04T09:20:54.6821920Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-12-04T09:20:54.6824079Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-12-04T09:20:54.6824503Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-12-04T09:20:54.6829371Z * [new tag] v1.9.1 -> v1.9.1 2025-12-04T09:20:54.6834092Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-12-04T09:20:54.6837747Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-12-04T09:20:54.6842223Z * [new tag] v2.0.0 -> v2.0.0 2025-12-04T09:20:54.6847537Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-12-04T09:20:54.6848082Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-12-04T09:20:54.6848378Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-12-04T09:20:54.6848650Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-12-04T09:20:54.6848912Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-12-04T09:20:54.6849233Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-12-04T09:20:54.6849510Z * [new tag] v2.0.1 -> v2.0.1 2025-12-04T09:20:54.6849774Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-12-04T09:20:54.6850027Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-12-04T09:20:54.6850286Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-12-04T09:20:54.6850544Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-12-04T09:20:54.6850800Z * [new tag] v2.1.0 -> v2.1.0 2025-12-04T09:20:54.6851066Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-12-04T09:20:54.6851328Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-12-04T09:20:54.6851843Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-12-04T09:20:54.6852106Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-12-04T09:20:54.6852369Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-12-04T09:20:54.6852632Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-12-04T09:20:54.6852891Z * [new tag] v2.1.1 -> v2.1.1 2025-12-04T09:20:54.6853142Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-12-04T09:20:54.6853403Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-12-04T09:20:54.6853667Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-12-04T09:20:54.6853930Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-12-04T09:20:54.6854181Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-12-04T09:20:54.6854438Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-12-04T09:20:54.6854696Z * [new tag] v2.1.2 -> v2.1.2 2025-12-04T09:20:54.6854951Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-12-04T09:20:54.6855209Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-12-04T09:20:54.6855465Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-12-04T09:20:54.6855718Z * [new tag] v2.2.0 -> v2.2.0 2025-12-04T09:20:54.6855961Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-12-04T09:20:54.6856218Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-12-04T09:20:54.6856480Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-12-04T09:20:54.6856732Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-12-04T09:20:54.6856991Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-12-04T09:20:54.6857247Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-12-04T09:20:54.6857509Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-12-04T09:20:54.6857763Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-12-04T09:20:54.6858016Z * [new tag] v2.2.1 -> v2.2.1 2025-12-04T09:20:54.6858269Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-12-04T09:20:54.6858520Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-12-04T09:20:54.6858784Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-12-04T09:20:54.6859108Z * [new tag] v2.2.2 -> v2.2.2 2025-12-04T09:20:54.6859370Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-12-04T09:20:54.6859632Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-12-04T09:20:54.6859901Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-12-04T09:20:54.6860207Z * [new tag] v2.3.0 -> v2.3.0 2025-12-04T09:20:54.6860464Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-12-04T09:20:54.6860741Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-12-04T09:20:54.6861021Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-12-04T09:20:54.6861295Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-12-04T09:20:54.6861557Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-12-04T09:20:54.6861823Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-12-04T09:20:54.6862089Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-12-04T09:20:54.6862350Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-12-04T09:20:54.6862603Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-12-04T09:20:54.6862874Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-12-04T09:20:54.6863153Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-12-04T09:20:54.6863410Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-12-04T09:20:54.6863674Z * [new tag] v2.3.1 -> v2.3.1 2025-12-04T09:20:54.6863934Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-12-04T09:20:54.6864198Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-12-04T09:20:54.6864457Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-12-04T09:20:54.6864720Z * [new tag] v2.4.0 -> v2.4.0 2025-12-04T09:20:54.6864984Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-12-04T09:20:54.6865255Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-12-04T09:20:54.6865502Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-12-04T09:20:54.6865756Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-12-04T09:20:54.6866010Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-12-04T09:20:54.6866255Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-12-04T09:20:54.6866507Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-12-04T09:20:54.6866758Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-12-04T09:20:54.6867008Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-12-04T09:20:54.6867257Z * [new tag] v2.4.1 -> v2.4.1 2025-12-04T09:20:54.6867508Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-12-04T09:20:54.6867760Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-12-04T09:20:54.6868013Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-12-04T09:20:54.6868263Z * [new tag] v2.5.0 -> v2.5.0 2025-12-04T09:20:54.6868516Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-12-04T09:20:54.6868772Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-12-04T09:20:54.6869024Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-12-04T09:20:54.6869274Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-12-04T09:20:54.6869527Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-12-04T09:20:54.6869783Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-12-04T09:20:54.6870082Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-12-04T09:20:54.6870339Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-12-04T09:20:54.6870590Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-12-04T09:20:54.6871076Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-12-04T09:20:54.6871429Z * [new tag] v2.5.1 -> v2.5.1 2025-12-04T09:20:54.6871706Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-12-04T09:20:54.6871972Z * [new tag] v2.6.0 -> v2.6.0 2025-12-04T09:20:54.6874536Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-12-04T09:20:54.6875026Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-12-04T09:20:54.6875346Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-12-04T09:20:54.6875969Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-12-04T09:20:54.6876617Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-12-04T09:20:54.6876946Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-12-04T09:20:54.6877396Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-12-04T09:20:54.6877706Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-12-04T09:20:54.6880239Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-12-04T09:20:54.6880785Z * [new tag] v2.7.0 -> v2.7.0 2025-12-04T09:20:54.6881204Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-12-04T09:20:54.6882109Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-12-04T09:20:54.6882444Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-12-04T09:20:54.6882740Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-12-04T09:20:54.6883014Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-12-04T09:20:54.6883603Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-12-04T09:20:54.6884102Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-12-04T09:20:54.6886694Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-12-04T09:20:54.6887036Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-12-04T09:20:54.6887300Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-12-04T09:20:54.6887572Z * [new tag] v2.7.1 -> v2.7.1 2025-12-04T09:20:54.6888019Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-12-04T09:20:54.6888315Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-12-04T09:20:54.6888701Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-12-04T09:20:54.6889875Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-12-04T09:20:54.6890170Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-12-04T09:20:54.6890618Z * [new tag] v2.8.0 -> v2.8.0 2025-12-04T09:20:54.6891626Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-12-04T09:20:54.6891978Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-12-04T09:20:54.6893524Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-12-04T09:20:54.6893822Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-12-04T09:20:54.6894384Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-12-04T09:20:54.6894916Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-12-04T09:20:54.6898492Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-12-04T09:20:54.6899063Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-12-04T09:20:54.6899334Z * [new tag] v2.9.0 -> v2.9.0 2025-12-04T09:20:54.6899594Z * [new tag] v2.9.0-rc1 -> v2.9.0-rc1 2025-12-04T09:20:54.6899951Z * [new tag] v2.9.0-rc10 -> v2.9.0-rc10 2025-12-04T09:20:54.6900230Z * [new tag] v2.9.0-rc11 -> v2.9.0-rc11 2025-12-04T09:20:54.6900671Z * [new tag] v2.9.0-rc2 -> v2.9.0-rc2 2025-12-04T09:20:54.6900958Z * [new tag] v2.9.0-rc3 -> v2.9.0-rc3 2025-12-04T09:20:54.6901318Z * [new tag] v2.9.0-rc4 -> v2.9.0-rc4 2025-12-04T09:20:54.6902421Z * [new tag] v2.9.0-rc5 -> v2.9.0-rc5 2025-12-04T09:20:54.6903028Z * [new tag] v2.9.0-rc6 -> v2.9.0-rc6 2025-12-04T09:20:54.6903711Z * [new tag] v2.9.0-rc7 -> v2.9.0-rc7 2025-12-04T09:20:54.6904727Z * [new tag] v2.9.0-rc8 -> v2.9.0-rc8 2025-12-04T09:20:54.6905119Z * [new tag] v2.9.0-rc9 -> v2.9.0-rc9 2025-12-04T09:20:54.6905614Z * [new tag] v2.9.1 -> v2.9.1 2025-12-04T09:20:54.6908836Z * [new tag] v2.9.1-rc1 -> v2.9.1-rc1 2025-12-04T09:20:54.6909155Z * [new tag] v2.9.1-rc2 -> v2.9.1-rc2 2025-12-04T09:20:54.6909438Z * [new tag] viable/strict/1759343184 -> viable/strict/1759343184 2025-12-04T09:20:54.6909748Z * [new tag] viable/strict/1759346540 -> viable/strict/1759346540 2025-12-04T09:20:54.6910046Z * [new tag] viable/strict/1759348181 -> viable/strict/1759348181 2025-12-04T09:20:54.6910340Z * [new tag] viable/strict/1759350324 -> viable/strict/1759350324 2025-12-04T09:20:54.6910999Z * [new tag] viable/strict/1759351793 -> viable/strict/1759351793 2025-12-04T09:20:54.6911379Z * [new tag] viable/strict/1759353844 -> viable/strict/1759353844 2025-12-04T09:20:54.6911854Z * [new tag] viable/strict/1759355374 -> viable/strict/1759355374 2025-12-04T09:20:54.6912295Z * [new tag] viable/strict/1759357472 -> viable/strict/1759357472 2025-12-04T09:20:54.6912722Z * [new tag] viable/strict/1759361002 -> viable/strict/1759361002 2025-12-04T09:20:54.6915071Z * [new tag] viable/strict/1759362585 -> viable/strict/1759362585 2025-12-04T09:20:54.6920018Z * [new tag] viable/strict/1759365359 -> viable/strict/1759365359 2025-12-04T09:20:54.6925526Z * [new tag] viable/strict/1759370089 -> viable/strict/1759370089 2025-12-04T09:20:54.6928415Z * [new tag] viable/strict/1759377554 -> viable/strict/1759377554 2025-12-04T09:20:54.6928855Z * [new tag] viable/strict/1759379133 -> viable/strict/1759379133 2025-12-04T09:20:54.6929188Z * [new tag] viable/strict/1759389871 -> viable/strict/1759389871 2025-12-04T09:20:54.6929511Z * [new tag] viable/strict/1759393562 -> viable/strict/1759393562 2025-12-04T09:20:54.6929844Z * [new tag] viable/strict/1759395076 -> viable/strict/1759395076 2025-12-04T09:20:54.6930166Z * [new tag] viable/strict/1759398579 -> viable/strict/1759398579 2025-12-04T09:20:54.6930488Z * [new tag] viable/strict/1759404142 -> viable/strict/1759404142 2025-12-04T09:20:54.6930806Z * [new tag] viable/strict/1759405773 -> viable/strict/1759405773 2025-12-04T09:20:54.6931126Z * [new tag] viable/strict/1759408041 -> viable/strict/1759408041 2025-12-04T09:20:54.6931665Z * [new tag] viable/strict/1759411593 -> viable/strict/1759411593 2025-12-04T09:20:54.6932195Z * [new tag] viable/strict/1759427395 -> viable/strict/1759427395 2025-12-04T09:20:54.6932512Z * [new tag] viable/strict/1759434582 -> viable/strict/1759434582 2025-12-04T09:20:54.6932848Z * [new tag] viable/strict/1759436720 -> viable/strict/1759436720 2025-12-04T09:20:54.6933238Z * [new tag] viable/strict/1759440219 -> viable/strict/1759440219 2025-12-04T09:20:54.6933559Z * [new tag] viable/strict/1759441948 -> viable/strict/1759441948 2025-12-04T09:20:54.6933887Z * [new tag] viable/strict/1759443860 -> viable/strict/1759443860 2025-12-04T09:20:54.6934203Z * [new tag] viable/strict/1759445377 -> viable/strict/1759445377 2025-12-04T09:20:54.6934528Z * [new tag] viable/strict/1759447415 -> viable/strict/1759447415 2025-12-04T09:20:54.6934834Z * [new tag] viable/strict/1759451750 -> viable/strict/1759451750 2025-12-04T09:20:54.6935162Z * [new tag] viable/strict/1759453910 -> viable/strict/1759453910 2025-12-04T09:20:54.6935467Z * [new tag] viable/strict/1759456483 -> viable/strict/1759456483 2025-12-04T09:20:54.6935755Z * [new tag] viable/strict/1759459279 -> viable/strict/1759459279 2025-12-04T09:20:54.6936071Z * [new tag] viable/strict/1759460742 -> viable/strict/1759460742 2025-12-04T09:20:54.6936420Z * [new tag] viable/strict/1759462025 -> viable/strict/1759462025 2025-12-04T09:20:54.6936726Z * [new tag] viable/strict/1759469086 -> viable/strict/1759469086 2025-12-04T09:20:54.6937009Z * [new tag] viable/strict/1759470581 -> viable/strict/1759470581 2025-12-04T09:20:54.6937305Z * [new tag] viable/strict/1759472786 -> viable/strict/1759472786 2025-12-04T09:20:54.6937592Z * [new tag] viable/strict/1759476294 -> viable/strict/1759476294 2025-12-04T09:20:54.6937886Z * [new tag] viable/strict/1759479963 -> viable/strict/1759479963 2025-12-04T09:20:54.6938167Z * [new tag] viable/strict/1759492177 -> viable/strict/1759492177 2025-12-04T09:20:54.6938459Z * [new tag] viable/strict/1759519278 -> viable/strict/1759519278 2025-12-04T09:20:54.6938748Z * [new tag] viable/strict/1759524580 -> viable/strict/1759524580 2025-12-04T09:20:54.6939037Z * [new tag] viable/strict/1759528193 -> viable/strict/1759528193 2025-12-04T09:20:54.6939319Z * [new tag] viable/strict/1759533797 -> viable/strict/1759533797 2025-12-04T09:20:54.6939646Z * [new tag] viable/strict/1759542780 -> viable/strict/1759542780 2025-12-04T09:20:54.6939941Z * [new tag] viable/strict/1759549779 -> viable/strict/1759549779 2025-12-04T09:20:54.6940304Z * [new tag] viable/strict/1759555455 -> viable/strict/1759555455 2025-12-04T09:20:54.6940763Z * [new tag] viable/strict/1759559176 -> viable/strict/1759559176 2025-12-04T09:20:54.6941231Z * [new tag] viable/strict/1759560629 -> viable/strict/1759560629 2025-12-04T09:20:54.6941664Z * [new tag] viable/strict/1759569848 -> viable/strict/1759569848 2025-12-04T09:20:54.6942101Z * [new tag] viable/strict/1759571382 -> viable/strict/1759571382 2025-12-04T09:20:54.6942516Z * [new tag] viable/strict/1759573474 -> viable/strict/1759573474 2025-12-04T09:20:54.6943329Z * [new tag] viable/strict/1759618187 -> viable/strict/1759618187 2025-12-04T09:20:54.6943846Z * [new tag] viable/strict/1759626742 -> viable/strict/1759626742 2025-12-04T09:20:54.6944354Z * [new tag] viable/strict/1759632427 -> viable/strict/1759632427 2025-12-04T09:20:54.6944662Z * [new tag] viable/strict/1759634971 -> viable/strict/1759634971 2025-12-04T09:20:54.6945343Z * [new tag] viable/strict/1759661382 -> viable/strict/1759661382 2025-12-04T09:20:54.6945634Z * [new tag] viable/strict/1759663294 -> viable/strict/1759663294 2025-12-04T09:20:54.6946074Z * [new tag] viable/strict/1759708178 -> viable/strict/1759708178 2025-12-04T09:20:54.6946469Z * [new tag] viable/strict/1759715695 -> viable/strict/1759715695 2025-12-04T09:20:54.6946898Z * [new tag] viable/strict/1759728293 -> viable/strict/1759728293 2025-12-04T09:20:54.6951118Z * [new tag] viable/strict/1759735513 -> viable/strict/1759735513 2025-12-04T09:20:54.6951643Z * [new tag] viable/strict/1759739177 -> viable/strict/1759739177 2025-12-04T09:20:54.6952080Z * [new tag] viable/strict/1759758635 -> viable/strict/1759758635 2025-12-04T09:20:54.6952516Z * [new tag] viable/strict/1759765784 -> viable/strict/1759765784 2025-12-04T09:20:54.6953257Z * [new tag] viable/strict/1759767948 -> viable/strict/1759767948 2025-12-04T09:20:54.6953644Z * [new tag] viable/strict/1759771461 -> viable/strict/1759771461 2025-12-04T09:20:54.6953986Z * [new tag] viable/strict/1759776706 -> viable/strict/1759776706 2025-12-04T09:20:54.6954336Z * [new tag] viable/strict/1759782317 -> viable/strict/1759782317 2025-12-04T09:20:54.6954660Z * [new tag] viable/strict/1759783777 -> viable/strict/1759783777 2025-12-04T09:20:54.6954979Z * [new tag] viable/strict/1759785815 -> viable/strict/1759785815 2025-12-04T09:20:54.6955314Z * [new tag] viable/strict/1759789459 -> viable/strict/1759789459 2025-12-04T09:20:54.6955690Z * [new tag] viable/strict/1759790974 -> viable/strict/1759790974 2025-12-04T09:20:54.6956022Z * [new tag] viable/strict/1759794583 -> viable/strict/1759794583 2025-12-04T09:20:54.6956346Z * [new tag] viable/strict/1759797408 -> viable/strict/1759797408 2025-12-04T09:20:54.6956678Z * [new tag] viable/strict/1759799518 -> viable/strict/1759799518 2025-12-04T09:20:54.6957005Z * [new tag] viable/strict/1759804909 -> viable/strict/1759804909 2025-12-04T09:20:54.6957342Z * [new tag] viable/strict/1759807643 -> viable/strict/1759807643 2025-12-04T09:20:54.6957661Z * [new tag] viable/strict/1759809089 -> viable/strict/1759809089 2025-12-04T09:20:54.6957981Z * [new tag] viable/strict/1759811145 -> viable/strict/1759811145 2025-12-04T09:20:54.6958307Z * [new tag] viable/strict/1759812581 -> viable/strict/1759812581 2025-12-04T09:20:54.6958631Z * [new tag] viable/strict/1759814683 -> viable/strict/1759814683 2025-12-04T09:20:54.6958957Z * [new tag] viable/strict/1759821889 -> viable/strict/1759821889 2025-12-04T09:20:54.6959295Z * [new tag] viable/strict/1759823376 -> viable/strict/1759823376 2025-12-04T09:20:54.6959610Z * [new tag] viable/strict/1759827107 -> viable/strict/1759827107 2025-12-04T09:20:54.6959928Z * [new tag] viable/strict/1759830577 -> viable/strict/1759830577 2025-12-04T09:20:54.6960231Z * [new tag] viable/strict/1759832720 -> viable/strict/1759832720 2025-12-04T09:20:54.6960526Z * [new tag] viable/strict/1759842063 -> viable/strict/1759842063 2025-12-04T09:20:54.6961134Z * [new tag] viable/strict/1759847121 -> viable/strict/1759847121 2025-12-04T09:20:54.6962170Z * [new tag] viable/strict/1759850721 -> viable/strict/1759850721 2025-12-04T09:20:54.6962702Z * [new tag] viable/strict/1759857870 -> viable/strict/1759857870 2025-12-04T09:20:54.6963253Z * [new tag] viable/strict/1759863143 -> viable/strict/1759863143 2025-12-04T09:20:54.6963793Z * [new tag] viable/strict/1759875874 -> viable/strict/1759875874 2025-12-04T09:20:54.6964327Z * [new tag] viable/strict/1759877385 -> viable/strict/1759877385 2025-12-04T09:20:54.6964969Z * [new tag] viable/strict/1759883801 -> viable/strict/1759883801 2025-12-04T09:20:54.6965654Z * [new tag] viable/strict/1759885922 -> viable/strict/1759885922 2025-12-04T09:20:54.6966221Z * [new tag] viable/strict/1759888488 -> viable/strict/1759888488 2025-12-04T09:20:54.6966820Z * [new tag] viable/strict/1759895471 -> viable/strict/1759895471 2025-12-04T09:20:54.6967501Z * [new tag] viable/strict/1759904803 -> viable/strict/1759904803 2025-12-04T09:20:54.6968549Z * [new tag] viable/strict/1759908300 -> viable/strict/1759908300 2025-12-04T09:20:54.6969288Z * [new tag] viable/strict/1759915520 -> viable/strict/1759915520 2025-12-04T09:20:54.6969888Z * [new tag] viable/strict/1759916978 -> viable/strict/1759916978 2025-12-04T09:20:54.6970228Z * [new tag] viable/strict/1759930024 -> viable/strict/1759930024 2025-12-04T09:20:54.6970759Z * [new tag] viable/strict/1759948122 -> viable/strict/1759948122 2025-12-04T09:20:54.6971435Z * [new tag] viable/strict/1759952983 -> viable/strict/1759952983 2025-12-04T09:20:54.6972488Z * [new tag] viable/strict/1759955121 -> viable/strict/1759955121 2025-12-04T09:20:54.6975410Z * [new tag] viable/strict/1759962298 -> viable/strict/1759962298 2025-12-04T09:20:54.6975796Z * [new tag] viable/strict/1759965837 -> viable/strict/1759965837 2025-12-04T09:20:54.6976122Z * [new tag] viable/strict/1759970213 -> viable/strict/1759970213 2025-12-04T09:20:54.6976430Z * [new tag] viable/strict/1759974894 -> viable/strict/1759974894 2025-12-04T09:20:54.6976772Z * [new tag] viable/strict/1759977763 -> viable/strict/1759977763 2025-12-04T09:20:54.6977256Z * [new tag] viable/strict/1759979241 -> viable/strict/1759979241 2025-12-04T09:20:54.6977940Z * [new tag] viable/strict/1759985417 -> viable/strict/1759985417 2025-12-04T09:20:54.6978362Z * [new tag] viable/strict/1759987490 -> viable/strict/1759987490 2025-12-04T09:20:54.6978691Z * [new tag] viable/strict/1759996180 -> viable/strict/1759996180 2025-12-04T09:20:54.6979008Z * [new tag] viable/strict/1760065682 -> viable/strict/1760065682 2025-12-04T09:20:54.6979323Z * [new tag] viable/strict/1760066894 -> viable/strict/1760066894 2025-12-04T09:20:54.6979822Z * [new tag] viable/strict/1760070345 -> viable/strict/1760070345 2025-12-04T09:20:54.6980405Z * [new tag] viable/strict/1760089782 -> viable/strict/1760089782 2025-12-04T09:20:54.6981081Z * [new tag] viable/strict/1760091921 -> viable/strict/1760091921 2025-12-04T09:20:54.6981733Z * [new tag] viable/strict/1760127924 -> viable/strict/1760127924 2025-12-04T09:20:54.6982356Z * [new tag] viable/strict/1760129489 -> viable/strict/1760129489 2025-12-04T09:20:54.6983007Z * [new tag] viable/strict/1760132980 -> viable/strict/1760132980 2025-12-04T09:20:54.6983708Z * [new tag] viable/strict/1760135060 -> viable/strict/1760135060 2025-12-04T09:20:54.6985378Z * [new tag] viable/strict/1760215782 -> viable/strict/1760215782 2025-12-04T09:20:54.6985872Z * [new tag] viable/strict/1760273849 -> viable/strict/1760273849 2025-12-04T09:20:54.6986357Z * [new tag] viable/strict/1760275517 -> viable/strict/1760275517 2025-12-04T09:20:54.6986828Z * [new tag] viable/strict/1760276979 -> viable/strict/1760276979 2025-12-04T09:20:54.6987550Z * [new tag] viable/strict/1760279007 -> viable/strict/1760279007 2025-12-04T09:20:54.6988025Z * [new tag] viable/strict/1760286328 -> viable/strict/1760286328 2025-12-04T09:20:54.6988837Z * [new tag] viable/strict/1760493304 -> viable/strict/1760493304 2025-12-04T09:20:54.6989592Z * [new tag] viable/strict/1760496298 -> viable/strict/1760496298 2025-12-04T09:20:54.6990088Z * [new tag] viable/strict/1760518396 -> viable/strict/1760518396 2025-12-04T09:20:54.6990531Z * [new tag] viable/strict/1760534864 -> viable/strict/1760534864 2025-12-04T09:20:54.6991265Z * [new tag] viable/strict/1760549062 -> viable/strict/1760549062 2025-12-04T09:20:54.6991640Z * [new tag] viable/strict/1760552799 -> viable/strict/1760552799 2025-12-04T09:20:54.6992003Z * [new tag] viable/strict/1760554355 -> viable/strict/1760554355 2025-12-04T09:20:54.6992582Z * [new tag] viable/strict/1760556275 -> viable/strict/1760556275 2025-12-04T09:20:54.6993197Z * [new tag] viable/strict/1760564979 -> viable/strict/1760564979 2025-12-04T09:20:54.6993796Z * [new tag] viable/strict/1760567049 -> viable/strict/1760567049 2025-12-04T09:20:54.6996726Z * [new tag] viable/strict/1760568585 -> viable/strict/1760568585 2025-12-04T09:20:54.6997271Z * [new tag] viable/strict/1760570630 -> viable/strict/1760570630 2025-12-04T09:20:54.6997617Z * [new tag] viable/strict/1760572180 -> viable/strict/1760572180 2025-12-04T09:20:54.6998062Z * [new tag] viable/strict/1760575094 -> viable/strict/1760575094 2025-12-04T09:20:54.6998817Z * [new tag] viable/strict/1760579709 -> viable/strict/1760579709 2025-12-04T09:20:54.6999191Z * [new tag] viable/strict/1760582614 -> viable/strict/1760582614 2025-12-04T09:20:54.6999525Z * [new tag] viable/strict/1760586815 -> viable/strict/1760586815 2025-12-04T09:20:54.6999832Z * [new tag] viable/strict/1760588829 -> viable/strict/1760588829 2025-12-04T09:20:54.7000146Z * [new tag] viable/strict/1760590200 -> viable/strict/1760590200 2025-12-04T09:20:54.7000775Z * [new tag] viable/strict/1760592311 -> viable/strict/1760592311 2025-12-04T09:20:54.7001347Z * [new tag] viable/strict/1760619733 -> viable/strict/1760619733 2025-12-04T09:20:54.7001854Z * [new tag] viable/strict/1760628335 -> viable/strict/1760628335 2025-12-04T09:20:54.7002649Z * [new tag] viable/strict/1760635490 -> viable/strict/1760635490 2025-12-04T09:20:54.7003146Z * [new tag] viable/strict/1760640743 -> viable/strict/1760640743 2025-12-04T09:20:54.7003737Z * [new tag] viable/strict/1760642528 -> viable/strict/1760642528 2025-12-04T09:20:54.7004283Z * [new tag] viable/strict/1760646330 -> viable/strict/1760646330 2025-12-04T09:20:54.7004905Z * [new tag] viable/strict/1760666101 -> viable/strict/1760666101 2025-12-04T09:20:54.7005541Z * [new tag] viable/strict/1760668990 -> viable/strict/1760668990 2025-12-04T09:20:54.7006246Z * [new tag] viable/strict/1760670600 -> viable/strict/1760670600 2025-12-04T09:20:54.7006817Z * [new tag] viable/strict/1760671704 -> viable/strict/1760671704 2025-12-04T09:20:54.7007458Z * [new tag] viable/strict/1760673121 -> viable/strict/1760673121 2025-12-04T09:20:54.7008038Z * [new tag] viable/strict/1760675352 -> viable/strict/1760675352 2025-12-04T09:20:54.7008673Z * [new tag] viable/strict/1760696731 -> viable/strict/1760696731 2025-12-04T09:20:54.7010745Z * [new tag] viable/strict/1760723515 -> viable/strict/1760723515 2025-12-04T09:20:54.7011214Z * [new tag] viable/strict/1760727234 -> viable/strict/1760727234 2025-12-04T09:20:54.7011877Z * [new tag] viable/strict/1760730578 -> viable/strict/1760730578 2025-12-04T09:20:54.7012425Z * [new tag] viable/strict/1760732726 -> viable/strict/1760732726 2025-12-04T09:20:54.7013156Z * [new tag] viable/strict/1760734180 -> viable/strict/1760734180 2025-12-04T09:20:54.7014002Z * [new tag] viable/strict/1760736251 -> viable/strict/1760736251 2025-12-04T09:20:54.7014329Z * [new tag] viable/strict/1760737772 -> viable/strict/1760737772 2025-12-04T09:20:54.7014913Z * [new tag] viable/strict/1760758005 -> viable/strict/1760758005 2025-12-04T09:20:54.7015515Z * [new tag] viable/strict/1760761532 -> viable/strict/1760761532 2025-12-04T09:20:54.7016228Z * [new tag] viable/strict/1760802581 -> viable/strict/1760802581 2025-12-04T09:20:54.7016809Z * [new tag] viable/strict/1760827772 -> viable/strict/1760827772 2025-12-04T09:20:54.7017414Z * [new tag] viable/strict/1760834524 -> viable/strict/1760834524 2025-12-04T09:20:54.7019552Z * [new tag] viable/strict/1760845009 -> viable/strict/1760845009 2025-12-04T09:20:54.7020118Z * [new tag] viable/strict/1760876836 -> viable/strict/1760876836 2025-12-04T09:20:54.7020590Z * [new tag] viable/strict/1760880329 -> viable/strict/1760880329 2025-12-04T09:20:54.7021071Z * [new tag] viable/strict/1760888987 -> viable/strict/1760888987 2025-12-04T09:20:54.7021525Z * [new tag] viable/strict/1760912664 -> viable/strict/1760912664 2025-12-04T09:20:54.7021853Z * [new tag] viable/strict/1760925321 -> viable/strict/1760925321 2025-12-04T09:20:54.7022179Z * [new tag] viable/strict/1760931488 -> viable/strict/1760931488 2025-12-04T09:20:54.7022796Z * [new tag] viable/strict/1760932693 -> viable/strict/1760932693 2025-12-04T09:20:54.7023214Z * [new tag] viable/strict/1761004184 -> viable/strict/1761004184 2025-12-04T09:20:54.7023852Z * [new tag] viable/strict/1761014748 -> viable/strict/1761014748 2025-12-04T09:20:54.7024610Z * [new tag] viable/strict/1761017491 -> viable/strict/1761017491 2025-12-04T09:20:54.7025306Z * [new tag] viable/strict/1761018806 -> viable/strict/1761018806 2025-12-04T09:20:54.7026020Z * [new tag] viable/strict/1761020754 -> viable/strict/1761020754 2025-12-04T09:20:54.7026817Z * [new tag] viable/strict/1761024303 -> viable/strict/1761024303 2025-12-04T09:20:54.7027433Z * [new tag] viable/strict/1761029582 -> viable/strict/1761029582 2025-12-04T09:20:54.7028033Z * [new tag] viable/strict/1761031535 -> viable/strict/1761031535 2025-12-04T09:20:54.7028757Z * [new tag] viable/strict/1761035196 -> viable/strict/1761035196 2025-12-04T09:20:54.7029367Z * [new tag] viable/strict/1761045825 -> viable/strict/1761045825 2025-12-04T09:20:54.7030026Z * [new tag] viable/strict/1761054796 -> viable/strict/1761054796 2025-12-04T09:20:54.7030557Z * [new tag] viable/strict/1761060314 -> viable/strict/1761060314 2025-12-04T09:20:54.7034136Z * [new tag] viable/strict/1761071198 -> viable/strict/1761071198 2025-12-04T09:20:54.7038940Z * [new tag] viable/strict/1761074628 -> viable/strict/1761074628 2025-12-04T09:20:54.7042284Z * [new tag] viable/strict/1761078351 -> viable/strict/1761078351 2025-12-04T09:20:54.7042634Z * [new tag] viable/strict/1761079822 -> viable/strict/1761079822 2025-12-04T09:20:54.7043107Z * [new tag] viable/strict/1761081873 -> viable/strict/1761081873 2025-12-04T09:20:54.7043413Z * [new tag] viable/strict/1761083392 -> viable/strict/1761083392 2025-12-04T09:20:54.7043730Z * [new tag] viable/strict/1761085465 -> viable/strict/1761085465 2025-12-04T09:20:54.7044041Z * [new tag] viable/strict/1761089099 -> viable/strict/1761089099 2025-12-04T09:20:54.7044415Z * [new tag] viable/strict/1761095535 -> viable/strict/1761095535 2025-12-04T09:20:54.7044718Z * [new tag] viable/strict/1761098119 -> viable/strict/1761098119 2025-12-04T09:20:54.7045024Z * [new tag] viable/strict/1761101330 -> viable/strict/1761101330 2025-12-04T09:20:54.7045331Z * [new tag] viable/strict/1761114425 -> viable/strict/1761114425 2025-12-04T09:20:54.7045659Z * [new tag] viable/strict/1761116036 -> viable/strict/1761116036 2025-12-04T09:20:54.7045971Z * [new tag] viable/strict/1761119379 -> viable/strict/1761119379 2025-12-04T09:20:54.7046278Z * [new tag] viable/strict/1761121601 -> viable/strict/1761121601 2025-12-04T09:20:54.7046586Z * [new tag] viable/strict/1761123234 -> viable/strict/1761123234 2025-12-04T09:20:54.7046890Z * [new tag] viable/strict/1761126621 -> viable/strict/1761126621 2025-12-04T09:20:54.7047197Z * [new tag] viable/strict/1761132259 -> viable/strict/1761132259 2025-12-04T09:20:54.7047647Z * [new tag] viable/strict/1761146746 -> viable/strict/1761146746 2025-12-04T09:20:54.7048535Z * [new tag] viable/strict/1761164752 -> viable/strict/1761164752 2025-12-04T09:20:54.7048921Z * [new tag] viable/strict/1761166198 -> viable/strict/1761166198 2025-12-04T09:20:54.7049264Z * [new tag] viable/strict/1761175424 -> viable/strict/1761175424 2025-12-04T09:20:54.7049600Z * [new tag] viable/strict/1761176983 -> viable/strict/1761176983 2025-12-04T09:20:54.7049917Z * [new tag] viable/strict/1761179891 -> viable/strict/1761179891 2025-12-04T09:20:54.7050236Z * [new tag] viable/strict/1761181930 -> viable/strict/1761181930 2025-12-04T09:20:54.7050548Z * [new tag] viable/strict/1761184516 -> viable/strict/1761184516 2025-12-04T09:20:54.7050857Z * [new tag] viable/strict/1761190179 -> viable/strict/1761190179 2025-12-04T09:20:54.7051165Z * [new tag] viable/strict/1761193558 -> viable/strict/1761193558 2025-12-04T09:20:54.7051776Z * [new tag] viable/strict/1761207990 -> viable/strict/1761207990 2025-12-04T09:20:54.7052115Z * [new tag] viable/strict/1761229539 -> viable/strict/1761229539 2025-12-04T09:20:54.7052417Z * [new tag] viable/strict/1761244031 -> viable/strict/1761244031 2025-12-04T09:20:54.7052733Z * [new tag] viable/strict/1761248986 -> viable/strict/1761248986 2025-12-04T09:20:54.7053027Z * [new tag] viable/strict/1761259791 -> viable/strict/1761259791 2025-12-04T09:20:54.7053340Z * [new tag] viable/strict/1761266139 -> viable/strict/1761266139 2025-12-04T09:20:54.7053653Z * [new tag] viable/strict/1761268316 -> viable/strict/1761268316 2025-12-04T09:20:54.7054149Z * [new tag] viable/strict/1761273805 -> viable/strict/1761273805 2025-12-04T09:20:54.7054578Z * [new tag] viable/strict/1761275261 -> viable/strict/1761275261 2025-12-04T09:20:54.7055024Z * [new tag] viable/strict/1761277913 -> viable/strict/1761277913 2025-12-04T09:20:54.7055552Z * [new tag] viable/strict/1761290701 -> viable/strict/1761290701 2025-12-04T09:20:54.7056183Z * [new tag] viable/strict/1761294396 -> viable/strict/1761294396 2025-12-04T09:20:54.7056759Z * [new tag] viable/strict/1761303047 -> viable/strict/1761303047 2025-12-04T09:20:54.7057315Z * [new tag] viable/strict/1761335388 -> viable/strict/1761335388 2025-12-04T09:20:54.7060177Z * [new tag] viable/strict/1761337551 -> viable/strict/1761337551 2025-12-04T09:20:54.7060720Z * [new tag] viable/strict/1761339007 -> viable/strict/1761339007 2025-12-04T09:20:54.7061055Z * [new tag] viable/strict/1761341050 -> viable/strict/1761341050 2025-12-04T09:20:54.7061375Z * [new tag] viable/strict/1761346188 -> viable/strict/1761346188 2025-12-04T09:20:54.7061692Z * [new tag] viable/strict/1761349792 -> viable/strict/1761349792 2025-12-04T09:20:54.7062229Z * [new tag] viable/strict/1761352620 -> viable/strict/1761352620 2025-12-04T09:20:54.7062689Z * [new tag] viable/strict/1761354730 -> viable/strict/1761354730 2025-12-04T09:20:54.7063366Z * [new tag] viable/strict/1761357298 -> viable/strict/1761357298 2025-12-04T09:20:54.7063740Z * [new tag] viable/strict/1761360201 -> viable/strict/1761360201 2025-12-04T09:20:54.7064069Z * [new tag] viable/strict/1761361753 -> viable/strict/1761361753 2025-12-04T09:20:54.7064770Z * [new tag] viable/strict/1761364351 -> viable/strict/1761364351 2025-12-04T09:20:54.7065100Z * [new tag] viable/strict/1761366338 -> viable/strict/1761366338 2025-12-04T09:20:54.7065419Z * [new tag] viable/strict/1761367802 -> viable/strict/1761367802 2025-12-04T09:20:54.7066020Z * [new tag] viable/strict/1761369889 -> viable/strict/1761369889 2025-12-04T09:20:54.7066621Z * [new tag] viable/strict/1761371385 -> viable/strict/1761371385 2025-12-04T09:20:54.7070478Z * [new tag] viable/strict/1761373581 -> viable/strict/1761373581 2025-12-04T09:20:54.7071026Z * [new tag] viable/strict/1761375054 -> viable/strict/1761375054 2025-12-04T09:20:54.7071467Z * [new tag] viable/strict/1761421785 -> viable/strict/1761421785 2025-12-04T09:20:54.7071908Z * [new tag] viable/strict/1761434614 -> viable/strict/1761434614 2025-12-04T09:20:54.7072222Z * [new tag] viable/strict/1761439254 -> viable/strict/1761439254 2025-12-04T09:20:54.7072687Z * [new tag] viable/strict/1761454187 -> viable/strict/1761454187 2025-12-04T09:20:54.7072991Z * [new tag] viable/strict/1761459991 -> viable/strict/1761459991 2025-12-04T09:20:54.7073273Z * [new tag] viable/strict/1761470668 -> viable/strict/1761470668 2025-12-04T09:20:54.7073685Z * [new tag] viable/strict/1761472188 -> viable/strict/1761472188 2025-12-04T09:20:54.7074089Z * [new tag] viable/strict/1761503178 -> viable/strict/1761503178 2025-12-04T09:20:54.7079211Z * [new tag] viable/strict/1761517492 -> viable/strict/1761517492 2025-12-04T09:20:54.7079720Z * [new tag] viable/strict/1761518981 -> viable/strict/1761518981 2025-12-04T09:20:54.7080050Z * [new tag] viable/strict/1761533609 -> viable/strict/1761533609 2025-12-04T09:20:54.7080538Z * [new tag] viable/strict/1761546438 -> viable/strict/1761546438 2025-12-04T09:20:54.7081256Z * [new tag] viable/strict/1761548133 -> viable/strict/1761548133 2025-12-04T09:20:54.7081791Z * [new tag] viable/strict/1761555186 -> viable/strict/1761555186 2025-12-04T09:20:54.7082213Z * [new tag] viable/strict/1761557178 -> viable/strict/1761557178 2025-12-04T09:20:54.7082934Z * [new tag] viable/strict/1761560772 -> viable/strict/1761560772 2025-12-04T09:20:54.7083307Z * [new tag] viable/strict/1761562266 -> viable/strict/1761562266 2025-12-04T09:20:54.7083813Z * [new tag] viable/strict/1761564260 -> viable/strict/1761564260 2025-12-04T09:20:54.7084129Z * [new tag] viable/strict/1761568072 -> viable/strict/1761568072 2025-12-04T09:20:54.7084439Z * [new tag] viable/strict/1761571683 -> viable/strict/1761571683 2025-12-04T09:20:54.7084835Z * [new tag] viable/strict/1761580199 -> viable/strict/1761580199 2025-12-04T09:20:54.7085140Z * [new tag] viable/strict/1761587383 -> viable/strict/1761587383 2025-12-04T09:20:54.7085448Z * [new tag] viable/strict/1761591165 -> viable/strict/1761591165 2025-12-04T09:20:54.7085759Z * [new tag] viable/strict/1761594575 -> viable/strict/1761594575 2025-12-04T09:20:54.7086060Z * [new tag] viable/strict/1761596710 -> viable/strict/1761596710 2025-12-04T09:20:54.7086381Z * [new tag] viable/strict/1761598189 -> viable/strict/1761598189 2025-12-04T09:20:54.7086697Z * [new tag] viable/strict/1761600254 -> viable/strict/1761600254 2025-12-04T09:20:54.7087007Z * [new tag] viable/strict/1761603879 -> viable/strict/1761603879 2025-12-04T09:20:54.7087313Z * [new tag] viable/strict/1761605429 -> viable/strict/1761605429 2025-12-04T09:20:54.7087623Z * [new tag] viable/strict/1761607468 -> viable/strict/1761607468 2025-12-04T09:20:54.7088113Z * [new tag] viable/strict/1761608983 -> viable/strict/1761608983 2025-12-04T09:20:54.7088918Z * [new tag] viable/strict/1761611846 -> viable/strict/1761611846 2025-12-04T09:20:54.7089310Z * [new tag] viable/strict/1761613922 -> viable/strict/1761613922 2025-12-04T09:20:54.7089641Z * [new tag] viable/strict/1761616504 -> viable/strict/1761616504 2025-12-04T09:20:54.7089959Z * [new tag] viable/strict/1761619599 -> viable/strict/1761619599 2025-12-04T09:20:54.7090306Z * [new tag] viable/strict/1761686693 -> viable/strict/1761686693 2025-12-04T09:20:54.7090614Z * [new tag] viable/strict/1761688179 -> viable/strict/1761688179 2025-12-04T09:20:54.7091284Z * [new tag] viable/strict/1761691973 -> viable/strict/1761691973 2025-12-04T09:20:54.7092073Z * [new tag] viable/strict/1761693884 -> viable/strict/1761693884 2025-12-04T09:20:54.7092593Z * [new tag] viable/strict/1761695389 -> viable/strict/1761695389 2025-12-04T09:20:54.7093265Z * [new tag] viable/strict/1761698408 -> viable/strict/1761698408 2025-12-04T09:20:54.7093848Z * [new tag] viable/strict/1761702931 -> viable/strict/1761702931 2025-12-04T09:20:54.7096336Z * [new tag] viable/strict/1761706307 -> viable/strict/1761706307 2025-12-04T09:20:54.7096714Z * [new tag] viable/strict/1761709065 -> viable/strict/1761709065 2025-12-04T09:20:54.7097061Z * [new tag] viable/strict/1761710285 -> viable/strict/1761710285 2025-12-04T09:20:54.7097368Z * [new tag] viable/strict/1761711983 -> viable/strict/1761711983 2025-12-04T09:20:54.7097688Z * [new tag] viable/strict/1761713514 -> viable/strict/1761713514 2025-12-04T09:20:54.7098020Z * [new tag] viable/strict/1761715523 -> viable/strict/1761715523 2025-12-04T09:20:54.7098775Z * [new tag] viable/strict/1761727973 -> viable/strict/1761727973 2025-12-04T09:20:54.7099291Z * [new tag] viable/strict/1761751558 -> viable/strict/1761751558 2025-12-04T09:20:54.7099970Z * [new tag] viable/strict/1761755187 -> viable/strict/1761755187 2025-12-04T09:20:54.7100667Z * [new tag] viable/strict/1761756826 -> viable/strict/1761756826 2025-12-04T09:20:54.7101337Z * [new tag] viable/strict/1761769551 -> viable/strict/1761769551 2025-12-04T09:20:54.7102056Z * [new tag] viable/strict/1761771032 -> viable/strict/1761771032 2025-12-04T09:20:54.7102607Z * [new tag] viable/strict/1761773101 -> viable/strict/1761773101 2025-12-04T09:20:54.7103266Z * [new tag] viable/strict/1761781792 -> viable/strict/1761781792 2025-12-04T09:20:54.7104026Z * [new tag] viable/strict/1761784788 -> viable/strict/1761784788 2025-12-04T09:20:54.7104584Z * [new tag] viable/strict/1761786740 -> viable/strict/1761786740 2025-12-04T09:20:54.7105328Z * [new tag] viable/strict/1761789332 -> viable/strict/1761789332 2025-12-04T09:20:54.7106572Z * [new tag] viable/strict/1761792569 -> viable/strict/1761792569 2025-12-04T09:20:54.7106892Z * [new tag] viable/strict/1761795289 -> viable/strict/1761795289 2025-12-04T09:20:54.7109782Z * [new tag] viable/strict/1761798345 -> viable/strict/1761798345 2025-12-04T09:20:54.7110200Z * [new tag] viable/strict/1761799827 -> viable/strict/1761799827 2025-12-04T09:20:54.7110527Z * [new tag] viable/strict/1761805604 -> viable/strict/1761805604 2025-12-04T09:20:54.7110992Z * [new tag] viable/strict/1761807202 -> viable/strict/1761807202 2025-12-04T09:20:54.7111788Z * [new tag] viable/strict/1761809094 -> viable/strict/1761809094 2025-12-04T09:20:54.7112327Z * [new tag] viable/strict/1761810576 -> viable/strict/1761810576 2025-12-04T09:20:54.7112768Z * [new tag] viable/strict/1761812771 -> viable/strict/1761812771 2025-12-04T09:20:54.7113528Z * [new tag] viable/strict/1761814363 -> viable/strict/1761814363 2025-12-04T09:20:54.7114079Z * [new tag] viable/strict/1761857410 -> viable/strict/1761857410 2025-12-04T09:20:54.7114545Z * [new tag] viable/strict/1761860985 -> viable/strict/1761860985 2025-12-04T09:20:54.7114897Z * [new tag] viable/strict/1761863094 -> viable/strict/1761863094 2025-12-04T09:20:54.7115224Z * [new tag] viable/strict/1761864590 -> viable/strict/1761864590 2025-12-04T09:20:54.7115855Z * [new tag] viable/strict/1761866675 -> viable/strict/1761866675 2025-12-04T09:20:54.7116839Z * [new tag] viable/strict/1761868178 -> viable/strict/1761868178 2025-12-04T09:20:54.7117641Z * [new tag] viable/strict/1761871111 -> viable/strict/1761871111 2025-12-04T09:20:54.7118143Z * [new tag] viable/strict/1761873126 -> viable/strict/1761873126 2025-12-04T09:20:54.7118566Z * [new tag] viable/strict/1761875714 -> viable/strict/1761875714 2025-12-04T09:20:54.7120024Z * [new tag] viable/strict/1761878924 -> viable/strict/1761878924 2025-12-04T09:20:54.7120406Z * [new tag] viable/strict/1761881727 -> viable/strict/1761881727 2025-12-04T09:20:54.7120870Z * [new tag] viable/strict/1761882959 -> viable/strict/1761882959 2025-12-04T09:20:54.7121210Z * [new tag] viable/strict/1761886268 -> viable/strict/1761886268 2025-12-04T09:20:54.7122146Z * [new tag] viable/strict/1761893641 -> viable/strict/1761893641 2025-12-04T09:20:54.7122626Z * [new tag] viable/strict/1761931517 -> viable/strict/1761931517 2025-12-04T09:20:54.7123257Z * [new tag] viable/strict/1761933080 -> viable/strict/1761933080 2025-12-04T09:20:54.7125483Z * [new tag] viable/strict/1761935217 -> viable/strict/1761935217 2025-12-04T09:20:54.7125871Z * [new tag] viable/strict/1761938533 -> viable/strict/1761938533 2025-12-04T09:20:54.7126209Z * [new tag] viable/strict/1761940184 -> viable/strict/1761940184 2025-12-04T09:20:54.7126554Z * [new tag] viable/strict/1761942338 -> viable/strict/1761942338 2025-12-04T09:20:54.7127062Z * [new tag] viable/strict/1761946100 -> viable/strict/1761946100 2025-12-04T09:20:54.7127435Z * [new tag] viable/strict/1761947374 -> viable/strict/1761947374 2025-12-04T09:20:54.7128072Z * [new tag] viable/strict/1761950978 -> viable/strict/1761950978 2025-12-04T09:20:54.7128746Z * [new tag] viable/strict/1761957727 -> viable/strict/1761957727 2025-12-04T09:20:54.7129361Z * [new tag] viable/strict/1761959532 -> viable/strict/1761959532 2025-12-04T09:20:54.7130044Z * [new tag] viable/strict/1761965366 -> viable/strict/1761965366 2025-12-04T09:20:54.7130810Z * [new tag] viable/strict/1761968066 -> viable/strict/1761968066 2025-12-04T09:20:54.7131424Z * [new tag] viable/strict/1761969322 -> viable/strict/1761969322 2025-12-04T09:20:54.7132460Z * [new tag] viable/strict/1761974723 -> viable/strict/1761974723 2025-12-04T09:20:54.7133393Z * [new tag] viable/strict/1761981837 -> viable/strict/1761981837 2025-12-04T09:20:54.7133538Z * [new tag] viable/strict/1761985546 -> viable/strict/1761985546 2025-12-04T09:20:54.7138594Z * [new tag] viable/strict/1761987030 -> viable/strict/1761987030 2025-12-04T09:20:54.7138790Z * [new tag] viable/strict/1762003554 -> viable/strict/1762003554 2025-12-04T09:20:54.7138920Z * [new tag] viable/strict/1762021560 -> viable/strict/1762021560 2025-12-04T09:20:54.7139047Z * [new tag] viable/strict/1762032190 -> viable/strict/1762032190 2025-12-04T09:20:54.7139188Z * [new tag] viable/strict/1762040981 -> viable/strict/1762040981 2025-12-04T09:20:54.7139314Z * [new tag] viable/strict/1762048525 -> viable/strict/1762048525 2025-12-04T09:20:54.7139446Z * [new tag] viable/strict/1762104223 -> viable/strict/1762104223 2025-12-04T09:20:54.7139752Z * [new tag] viable/strict/1762105778 -> viable/strict/1762105778 2025-12-04T09:20:54.7139962Z * [new tag] viable/strict/1762115109 -> viable/strict/1762115109 2025-12-04T09:20:54.7140634Z * [new tag] viable/strict/1762125840 -> viable/strict/1762125840 2025-12-04T09:20:54.7141099Z * [new tag] viable/strict/1762127377 -> viable/strict/1762127377 2025-12-04T09:20:54.7143765Z * [new tag] viable/strict/1762134925 -> viable/strict/1762134925 2025-12-04T09:20:54.7144062Z * [new tag] viable/strict/1762138338 -> viable/strict/1762138338 2025-12-04T09:20:54.7144504Z * [new tag] viable/strict/1762148993 -> viable/strict/1762148993 2025-12-04T09:20:54.7144663Z * [new tag] viable/strict/1762152871 -> viable/strict/1762152871 2025-12-04T09:20:54.7144809Z * [new tag] viable/strict/1762156183 -> viable/strict/1762156183 2025-12-04T09:20:54.7149246Z * [new tag] viable/strict/1762163457 -> viable/strict/1762163457 2025-12-04T09:20:54.7149478Z * [new tag] viable/strict/1762165569 -> viable/strict/1762165569 2025-12-04T09:20:54.7149633Z * [new tag] viable/strict/1762169035 -> viable/strict/1762169035 2025-12-04T09:20:54.7149782Z * [new tag] viable/strict/1762174936 -> viable/strict/1762174936 2025-12-04T09:20:54.7149965Z * [new tag] viable/strict/1762194412 -> viable/strict/1762194412 2025-12-04T09:20:54.7150162Z * [new tag] viable/strict/1762195876 -> viable/strict/1762195876 2025-12-04T09:20:54.7150310Z * [new tag] viable/strict/1762197788 -> viable/strict/1762197788 2025-12-04T09:20:54.7150467Z * [new tag] viable/strict/1762199389 -> viable/strict/1762199389 2025-12-04T09:20:54.7156084Z * [new tag] viable/strict/1762206585 -> viable/strict/1762206585 2025-12-04T09:20:54.7160844Z * [new tag] viable/strict/1762210184 -> viable/strict/1762210184 2025-12-04T09:20:54.7162801Z * [new tag] viable/strict/1762218736 -> viable/strict/1762218736 2025-12-04T09:20:54.7163025Z * [new tag] viable/strict/1762224529 -> viable/strict/1762224529 2025-12-04T09:20:54.7163288Z * [new tag] viable/strict/1762227253 -> viable/strict/1762227253 2025-12-04T09:20:54.7163408Z * [new tag] viable/strict/1762228515 -> viable/strict/1762228515 2025-12-04T09:20:54.7163535Z * [new tag] viable/strict/1762230349 -> viable/strict/1762230349 2025-12-04T09:20:54.7163652Z * [new tag] viable/strict/1762231859 -> viable/strict/1762231859 2025-12-04T09:20:54.7163777Z * [new tag] viable/strict/1762233925 -> viable/strict/1762233925 2025-12-04T09:20:54.7163893Z * [new tag] viable/strict/1762237630 -> viable/strict/1762237630 2025-12-04T09:20:54.7164017Z * [new tag] viable/strict/1762253522 -> viable/strict/1762253522 2025-12-04T09:20:54.7164139Z * [new tag] viable/strict/1762278588 -> viable/strict/1762278588 2025-12-04T09:20:54.7164253Z * [new tag] viable/strict/1762284203 -> viable/strict/1762284203 2025-12-04T09:20:54.7164378Z * [new tag] viable/strict/1762289446 -> viable/strict/1762289446 2025-12-04T09:20:54.7164492Z * [new tag] viable/strict/1762291515 -> viable/strict/1762291515 2025-12-04T09:20:54.7164608Z * [new tag] viable/strict/1762295100 -> viable/strict/1762295100 2025-12-04T09:20:54.7164731Z * [new tag] viable/strict/1762296590 -> viable/strict/1762296590 2025-12-04T09:20:54.7164845Z * [new tag] viable/strict/1762300179 -> viable/strict/1762300179 2025-12-04T09:20:54.7164962Z * [new tag] viable/strict/1762303207 -> viable/strict/1762303207 2025-12-04T09:20:54.7167516Z * [new tag] viable/strict/1762386584 -> viable/strict/1762386584 2025-12-04T09:20:54.7167654Z * [new tag] viable/strict/1762391537 -> viable/strict/1762391537 2025-12-04T09:20:54.7167798Z * [new tag] viable/strict/1762394119 -> viable/strict/1762394119 2025-12-04T09:20:54.7167974Z * [new tag] viable/strict/1762397437 -> viable/strict/1762397437 2025-12-04T09:20:54.7168116Z * [new tag] viable/strict/1762400256 -> viable/strict/1762400256 2025-12-04T09:20:54.7168239Z * [new tag] viable/strict/1762401469 -> viable/strict/1762401469 2025-12-04T09:20:54.7168395Z * [new tag] viable/strict/1762408195 -> viable/strict/1762408195 2025-12-04T09:20:54.7168679Z * [new tag] viable/strict/1762410411 -> viable/strict/1762410411 2025-12-04T09:20:54.7178717Z * [new tag] viable/strict/1762417613 -> viable/strict/1762417613 2025-12-04T09:20:54.7181267Z * [new tag] viable/strict/1762419198 -> viable/strict/1762419198 2025-12-04T09:20:54.7181564Z * [new tag] viable/strict/1762422656 -> viable/strict/1762422656 2025-12-04T09:20:54.7181780Z * [new tag] viable/strict/1762424746 -> viable/strict/1762424746 2025-12-04T09:20:54.7181936Z * [new tag] viable/strict/1762446386 -> viable/strict/1762446386 2025-12-04T09:20:54.7182148Z * [new tag] viable/strict/1762449912 -> viable/strict/1762449912 2025-12-04T09:20:54.7188169Z * [new tag] viable/strict/1762457031 -> viable/strict/1762457031 2025-12-04T09:20:54.7193313Z * [new tag] viable/strict/1762462441 -> viable/strict/1762462441 2025-12-04T09:20:54.7197690Z * [new tag] viable/strict/1762467909 -> viable/strict/1762467909 2025-12-04T09:20:54.7202015Z * [new tag] viable/strict/1762471493 -> viable/strict/1762471493 2025-12-04T09:20:54.7206354Z * [new tag] viable/strict/1762475990 -> viable/strict/1762475990 2025-12-04T09:20:54.7206536Z * [new tag] viable/strict/1762477933 -> viable/strict/1762477933 2025-12-04T09:20:54.7206681Z * [new tag] viable/strict/1762491053 -> viable/strict/1762491053 2025-12-04T09:20:54.7207021Z * [new tag] viable/strict/1762493118 -> viable/strict/1762493118 2025-12-04T09:20:54.7207170Z * [new tag] viable/strict/1762498442 -> viable/strict/1762498442 2025-12-04T09:20:54.7207298Z * [new tag] viable/strict/1762501778 -> viable/strict/1762501778 2025-12-04T09:20:54.7207432Z * [new tag] viable/strict/1762504001 -> viable/strict/1762504001 2025-12-04T09:20:54.7207556Z * [new tag] viable/strict/1762505583 -> viable/strict/1762505583 2025-12-04T09:20:54.7207680Z * [new tag] viable/strict/1762507523 -> viable/strict/1762507523 2025-12-04T09:20:54.7207827Z * [new tag] viable/strict/1762511140 -> viable/strict/1762511140 2025-12-04T09:20:54.7207948Z * [new tag] viable/strict/1762512632 -> viable/strict/1762512632 2025-12-04T09:20:54.7208076Z * [new tag] viable/strict/1762520467 -> viable/strict/1762520467 2025-12-04T09:20:54.7208201Z * [new tag] viable/strict/1762522016 -> viable/strict/1762522016 2025-12-04T09:20:54.7208321Z * [new tag] viable/strict/1762530591 -> viable/strict/1762530591 2025-12-04T09:20:54.7208454Z * [new tag] viable/strict/1762543405 -> viable/strict/1762543405 2025-12-04T09:20:54.7208578Z * [new tag] viable/strict/1762544998 -> viable/strict/1762544998 2025-12-04T09:20:54.7208698Z * [new tag] viable/strict/1762552182 -> viable/strict/1762552182 2025-12-04T09:20:54.7208825Z * [new tag] viable/strict/1762554297 -> viable/strict/1762554297 2025-12-04T09:20:54.7208947Z * [new tag] viable/strict/1762559381 -> viable/strict/1762559381 2025-12-04T09:20:54.7209075Z * [new tag] viable/strict/1762562222 -> viable/strict/1762562222 2025-12-04T09:20:54.7209194Z * [new tag] viable/strict/1762564319 -> viable/strict/1762564319 2025-12-04T09:20:54.7209314Z * [new tag] viable/strict/1762566904 -> viable/strict/1762566904 2025-12-04T09:20:54.7209442Z * [new tag] viable/strict/1762569781 -> viable/strict/1762569781 2025-12-04T09:20:54.7209560Z * [new tag] viable/strict/1762575940 -> viable/strict/1762575940 2025-12-04T09:20:54.7209688Z * [new tag] viable/strict/1762580974 -> viable/strict/1762580974 2025-12-04T09:20:54.7209806Z * [new tag] viable/strict/1762583185 -> viable/strict/1762583185 2025-12-04T09:20:54.7209926Z * [new tag] viable/strict/1762586647 -> viable/strict/1762586647 2025-12-04T09:20:54.7210057Z * [new tag] viable/strict/1762588183 -> viable/strict/1762588183 2025-12-04T09:20:54.7210180Z * [new tag] viable/strict/1762593886 -> viable/strict/1762593886 2025-12-04T09:20:54.7210298Z * [new tag] viable/strict/1762650743 -> viable/strict/1762650743 2025-12-04T09:20:54.7210439Z * [new tag] viable/strict/1762653328 -> viable/strict/1762653328 2025-12-04T09:20:54.7210560Z * [new tag] viable/strict/1762659342 -> viable/strict/1762659342 2025-12-04T09:20:54.7210688Z * [new tag] viable/strict/1762662360 -> viable/strict/1762662360 2025-12-04T09:20:54.7210808Z * [new tag] viable/strict/1762667377 -> viable/strict/1762667377 2025-12-04T09:20:54.7210941Z * [new tag] viable/strict/1762671090 -> viable/strict/1762671090 2025-12-04T09:20:54.7211680Z * [new tag] viable/strict/1762680284 -> viable/strict/1762680284 2025-12-04T09:20:54.7216003Z * [new tag] viable/strict/1762683900 -> viable/strict/1762683900 2025-12-04T09:20:54.7216160Z * [new tag] viable/strict/1762705541 -> viable/strict/1762705541 2025-12-04T09:20:54.7216355Z * [new tag] viable/strict/1762709004 -> viable/strict/1762709004 2025-12-04T09:20:54.7216646Z * [new tag] viable/strict/1762746004 -> viable/strict/1762746004 2025-12-04T09:20:54.7216787Z * [new tag] viable/strict/1762748799 -> viable/strict/1762748799 2025-12-04T09:20:54.7216917Z * [new tag] viable/strict/1762759504 -> viable/strict/1762759504 2025-12-04T09:20:54.7219965Z * [new tag] viable/strict/1762760973 -> viable/strict/1762760973 2025-12-04T09:20:54.7223328Z * [new tag] viable/strict/1762775374 -> viable/strict/1762775374 2025-12-04T09:20:54.7223901Z * [new tag] viable/strict/1762777661 -> viable/strict/1762777661 2025-12-04T09:20:54.7224085Z * [new tag] viable/strict/1762779774 -> viable/strict/1762779774 2025-12-04T09:20:54.7224215Z * [new tag] viable/strict/1762781259 -> viable/strict/1762781259 2025-12-04T09:20:54.7224372Z * [new tag] viable/strict/1762793628 -> viable/strict/1762793628 2025-12-04T09:20:54.7224527Z * [new tag] viable/strict/1762800711 -> viable/strict/1762800711 2025-12-04T09:20:54.7224661Z * [new tag] viable/strict/1762809894 -> viable/strict/1762809894 2025-12-04T09:20:54.7224806Z * [new tag] viable/strict/1762811384 -> viable/strict/1762811384 2025-12-04T09:20:54.7224953Z * [new tag] viable/strict/1762813841 -> viable/strict/1762813841 2025-12-04T09:20:54.7229724Z * [new tag] viable/strict/1762815047 -> viable/strict/1762815047 2025-12-04T09:20:54.7233803Z * [new tag] viable/strict/1762817094 -> viable/strict/1762817094 2025-12-04T09:20:54.7238376Z * [new tag] viable/strict/1762818582 -> viable/strict/1762818582 2025-12-04T09:20:54.7243277Z * [new tag] viable/strict/1762821623 -> viable/strict/1762821623 2025-12-04T09:20:54.7245414Z * [new tag] viable/strict/1762823531 -> viable/strict/1762823531 2025-12-04T09:20:54.7245604Z * [new tag] viable/strict/1762849583 -> viable/strict/1762849583 2025-12-04T09:20:54.7245741Z * [new tag] viable/strict/1762851200 -> viable/strict/1762851200 2025-12-04T09:20:54.7245869Z * [new tag] viable/strict/1762854603 -> viable/strict/1762854603 2025-12-04T09:20:54.7246011Z * [new tag] viable/strict/1762858276 -> viable/strict/1762858276 2025-12-04T09:20:54.7246142Z * [new tag] viable/strict/1762860891 -> viable/strict/1762860891 2025-12-04T09:20:54.7246277Z * [new tag] viable/strict/1762866174 -> viable/strict/1762866174 2025-12-04T09:20:54.7246416Z * [new tag] viable/strict/1762867653 -> viable/strict/1762867653 2025-12-04T09:20:54.7246543Z * [new tag] viable/strict/1762872669 -> viable/strict/1762872669 2025-12-04T09:20:54.7246680Z * [new tag] viable/strict/1762878380 -> viable/strict/1762878380 2025-12-04T09:20:54.7246814Z * [new tag] viable/strict/1762889003 -> viable/strict/1762889003 2025-12-04T09:20:54.7246943Z * [new tag] viable/strict/1762890589 -> viable/strict/1762890589 2025-12-04T09:20:54.7247069Z * [new tag] viable/strict/1762892743 -> viable/strict/1762892743 2025-12-04T09:20:54.7247207Z * [new tag] viable/strict/1762894271 -> viable/strict/1762894271 2025-12-04T09:20:54.7247346Z * [new tag] viable/strict/1762896287 -> viable/strict/1762896287 2025-12-04T09:20:54.7247482Z * [new tag] viable/strict/1762915871 -> viable/strict/1762915871 2025-12-04T09:20:54.7247764Z * [new tag] viable/strict/1762918569 -> viable/strict/1762918569 2025-12-04T09:20:54.7247909Z * [new tag] viable/strict/1762919776 -> viable/strict/1762919776 2025-12-04T09:20:54.7248044Z * [new tag] viable/strict/1762923072 -> viable/strict/1762923072 2025-12-04T09:20:54.7248237Z * [new tag] viable/strict/1762928826 -> viable/strict/1762928826 2025-12-04T09:20:54.7248365Z * [new tag] viable/strict/1762930451 -> viable/strict/1762930451 2025-12-04T09:20:54.7248501Z * [new tag] viable/strict/1762933780 -> viable/strict/1762933780 2025-12-04T09:20:54.7248646Z * [new tag] viable/strict/1762937638 -> viable/strict/1762937638 2025-12-04T09:20:54.7248796Z * [new tag] viable/strict/1762939545 -> viable/strict/1762939545 2025-12-04T09:20:54.7248936Z * [new tag] viable/strict/1762962692 -> viable/strict/1762962692 2025-12-04T09:20:54.7249060Z * [new tag] viable/strict/1762979143 -> viable/strict/1762979143 2025-12-04T09:20:54.7249194Z * [new tag] viable/strict/1762984188 -> viable/strict/1762984188 2025-12-04T09:20:54.7249337Z * [new tag] viable/strict/1762986306 -> viable/strict/1762986306 2025-12-04T09:20:54.7249480Z * [new tag] viable/strict/1762989903 -> viable/strict/1762989903 2025-12-04T09:20:54.7249609Z * [new tag] viable/strict/1762991377 -> viable/strict/1762991377 2025-12-04T09:20:54.7249744Z * [new tag] viable/strict/1762998921 -> viable/strict/1762998921 2025-12-04T09:20:54.7249883Z * [new tag] viable/strict/1763002287 -> viable/strict/1763002287 2025-12-04T09:20:54.7250178Z * [new tag] viable/strict/1763016840 -> viable/strict/1763016840 2025-12-04T09:20:54.7251206Z * [new tag] viable/strict/1763020180 -> viable/strict/1763020180 2025-12-04T09:20:54.7251778Z * [new tag] viable/strict/1763027421 -> viable/strict/1763027421 2025-12-04T09:20:54.7253129Z * [new tag] viable/strict/1763031120 -> viable/strict/1763031120 2025-12-04T09:20:54.7253269Z * [new tag] viable/strict/1763036861 -> viable/strict/1763036861 2025-12-04T09:20:54.7256843Z * [new tag] viable/strict/1763038993 -> viable/strict/1763038993 2025-12-04T09:20:54.7257010Z * [new tag] viable/strict/1763054703 -> viable/strict/1763054703 2025-12-04T09:20:54.7257163Z * [new tag] viable/strict/1763067061 -> viable/strict/1763067061 2025-12-04T09:20:54.7257303Z * [new tag] viable/strict/1763070847 -> viable/strict/1763070847 2025-12-04T09:20:54.7257449Z * [new tag] viable/strict/1763072706 -> viable/strict/1763072706 2025-12-04T09:20:54.7258047Z * [new tag] viable/strict/1763076302 -> viable/strict/1763076302 2025-12-04T09:20:54.7258816Z * [new tag] viable/strict/1763080816 -> viable/strict/1763080816 2025-12-04T09:20:54.7259331Z * [new tag] viable/strict/1763082732 -> viable/strict/1763082732 2025-12-04T09:20:54.7260171Z * [new tag] viable/strict/1763085329 -> viable/strict/1763085329 2025-12-04T09:20:54.7260919Z * [new tag] viable/strict/1763088623 -> viable/strict/1763088623 2025-12-04T09:20:54.7261567Z * [new tag] viable/strict/1763091402 -> viable/strict/1763091402 2025-12-04T09:20:54.7262824Z * [new tag] viable/strict/1763092602 -> viable/strict/1763092602 2025-12-04T09:20:54.7262956Z * [new tag] viable/strict/1763094355 -> viable/strict/1763094355 2025-12-04T09:20:54.7263700Z * [new tag] viable/strict/1763099390 -> viable/strict/1763099390 2025-12-04T09:20:54.7264340Z * [new tag] viable/strict/1763101608 -> viable/strict/1763101608 2025-12-04T09:20:54.7264990Z * [new tag] viable/strict/1763105102 -> viable/strict/1763105102 2025-12-04T09:20:54.7269018Z * [new tag] viable/strict/1763112347 -> viable/strict/1763112347 2025-12-04T09:20:54.7269172Z * [new tag] viable/strict/1763119471 -> viable/strict/1763119471 2025-12-04T09:20:54.7269451Z * [new tag] viable/strict/1763126835 -> viable/strict/1763126835 2025-12-04T09:20:54.7269574Z * [new tag] viable/strict/1763149779 -> viable/strict/1763149779 2025-12-04T09:20:54.7269693Z * [new tag] viable/strict/1763164178 -> viable/strict/1763164178 2025-12-04T09:20:54.7269815Z * [new tag] viable/strict/1763167104 -> viable/strict/1763167104 2025-12-04T09:20:54.7269938Z * [new tag] viable/strict/1763169132 -> viable/strict/1763169132 2025-12-04T09:20:54.7270276Z * [new tag] viable/strict/1763171708 -> viable/strict/1763171708 2025-12-04T09:20:54.7271030Z * [new tag] viable/strict/1763174759 -> viable/strict/1763174759 2025-12-04T09:20:54.7271687Z * [new tag] viable/strict/1763180744 -> viable/strict/1763180744 2025-12-04T09:20:54.7272552Z * [new tag] viable/strict/1763182227 -> viable/strict/1763182227 2025-12-04T09:20:54.7273087Z * [new tag] viable/strict/1763184309 -> viable/strict/1763184309 2025-12-04T09:20:54.7274359Z * [new tag] viable/strict/1763187991 -> viable/strict/1763187991 2025-12-04T09:20:54.7274775Z * [new tag] viable/strict/1763191445 -> viable/strict/1763191445 2025-12-04T09:20:54.7275782Z * [new tag] viable/strict/1763195152 -> viable/strict/1763195152 2025-12-04T09:20:54.7276021Z * [new tag] viable/strict/1763205769 -> viable/strict/1763205769 2025-12-04T09:20:54.7277244Z * [new tag] viable/strict/1763246990 -> viable/strict/1763246990 2025-12-04T09:20:54.7277738Z * [new tag] viable/strict/1763261578 -> viable/strict/1763261578 2025-12-04T09:20:54.7278554Z * [new tag] viable/strict/1763286573 -> viable/strict/1763286573 2025-12-04T09:20:54.7278846Z * [new tag] viable/strict/1763292167 -> viable/strict/1763292167 2025-12-04T09:20:54.7279835Z * [new tag] viable/strict/1763333386 -> viable/strict/1763333386 2025-12-04T09:20:54.7280056Z * [new tag] viable/strict/1763340082 -> viable/strict/1763340082 2025-12-04T09:20:54.7281752Z * [new tag] viable/strict/1763364324 -> viable/strict/1763364324 2025-12-04T09:20:54.7282203Z * [new tag] viable/strict/1763371569 -> viable/strict/1763371569 2025-12-04T09:20:54.7283141Z * [new tag] viable/strict/1763373067 -> viable/strict/1763373067 2025-12-04T09:20:54.7283618Z * [new tag] viable/strict/1763375157 -> viable/strict/1763375157 2025-12-04T09:20:54.7284537Z * [new tag] viable/strict/1763382462 -> viable/strict/1763382462 2025-12-04T09:20:54.7285071Z * [new tag] viable/strict/1763394661 -> viable/strict/1763394661 2025-12-04T09:20:54.7286122Z * [new tag] viable/strict/1763396797 -> viable/strict/1763396797 2025-12-04T09:20:54.7286395Z * [new tag] viable/strict/1763398542 -> viable/strict/1763398542 2025-12-04T09:20:54.7287516Z * [new tag] viable/strict/1763401807 -> viable/strict/1763401807 2025-12-04T09:20:54.7287812Z * [new tag] viable/strict/1763414698 -> viable/strict/1763414698 2025-12-04T09:20:54.7288782Z * [new tag] viable/strict/1763419807 -> viable/strict/1763419807 2025-12-04T09:20:54.7289298Z * [new tag] viable/strict/1763426369 -> viable/strict/1763426369 2025-12-04T09:20:54.7290197Z * [new tag] viable/strict/1763428331 -> viable/strict/1763428331 2025-12-04T09:20:54.7290719Z * [new tag] viable/strict/1763430922 -> viable/strict/1763430922 2025-12-04T09:20:54.7291636Z * [new tag] viable/strict/1763434184 -> viable/strict/1763434184 2025-12-04T09:20:54.7292078Z * [new tag] viable/strict/1763439973 -> viable/strict/1763439973 2025-12-04T09:20:54.7293390Z * [new tag] viable/strict/1763444995 -> viable/strict/1763444995 2025-12-04T09:20:54.7293530Z * [new tag] viable/strict/1763447206 -> viable/strict/1763447206 2025-12-04T09:20:54.7294538Z * [new tag] viable/strict/1763448826 -> viable/strict/1763448826 2025-12-04T09:20:54.7295003Z * [new tag] viable/strict/1763450717 -> viable/strict/1763450717 2025-12-04T09:20:54.7296005Z * [new tag] viable/strict/1763452183 -> viable/strict/1763452183 2025-12-04T09:20:54.7299188Z * [new tag] viable/strict/1763457945 -> viable/strict/1763457945 2025-12-04T09:20:54.7299327Z * [new tag] viable/strict/1763459439 -> viable/strict/1763459439 2025-12-04T09:20:54.7299450Z * [new tag] viable/strict/1763461556 -> viable/strict/1763461556 2025-12-04T09:20:54.7299579Z * [new tag] viable/strict/1763463103 -> viable/strict/1763463103 2025-12-04T09:20:54.7299699Z * [new tag] viable/strict/1763465100 -> viable/strict/1763465100 2025-12-04T09:20:54.7303004Z * [new tag] viable/strict/1763468866 -> viable/strict/1763468866 2025-12-04T09:20:54.7303148Z * [new tag] viable/strict/1763493823 -> viable/strict/1763493823 2025-12-04T09:20:54.7303413Z * [new tag] viable/strict/1763496249 -> viable/strict/1763496249 2025-12-04T09:20:54.7303908Z * [new tag] viable/strict/1763502620 -> viable/strict/1763502620 2025-12-04T09:20:54.7304072Z * [new tag] viable/strict/1763504715 -> viable/strict/1763504715 2025-12-04T09:20:54.7309318Z * [new tag] viable/strict/1763506208 -> viable/strict/1763506208 2025-12-04T09:20:54.7312809Z * [new tag] viable/strict/1763520590 -> viable/strict/1763520590 2025-12-04T09:20:54.7314723Z * [new tag] viable/strict/1763523357 -> viable/strict/1763523357 2025-12-04T09:20:54.7314887Z * [new tag] viable/strict/1763529922 -> viable/strict/1763529922 2025-12-04T09:20:54.7315027Z * [new tag] viable/strict/1763531408 -> viable/strict/1763531408 2025-12-04T09:20:54.7315189Z * [new tag] viable/strict/1763533622 -> viable/strict/1763533622 2025-12-04T09:20:54.7315358Z * [new tag] viable/strict/1763538576 -> viable/strict/1763538576 2025-12-04T09:20:54.7315499Z * [new tag] viable/strict/1763545823 -> viable/strict/1763545823 2025-12-04T09:20:54.7315615Z * [new tag] viable/strict/1763547951 -> viable/strict/1763547951 2025-12-04T09:20:54.7315748Z * [new tag] viable/strict/1763551477 -> viable/strict/1763551477 2025-12-04T09:20:54.7315865Z * [new tag] viable/strict/1763552982 -> viable/strict/1763552982 2025-12-04T09:20:54.7315987Z * [new tag] viable/strict/1763594698 -> viable/strict/1763594698 2025-12-04T09:20:54.7316106Z * [new tag] viable/strict/1763596178 -> viable/strict/1763596178 2025-12-04T09:20:54.7316222Z * [new tag] viable/strict/1763599155 -> viable/strict/1763599155 2025-12-04T09:20:54.7316348Z * [new tag] viable/strict/1763603717 -> viable/strict/1763603717 2025-12-04T09:20:54.7316482Z * [new tag] viable/strict/1763606923 -> viable/strict/1763606923 2025-12-04T09:20:54.7316600Z * [new tag] viable/strict/1763609715 -> viable/strict/1763609715 2025-12-04T09:20:54.7316741Z * [new tag] viable/strict/1763612757 -> viable/strict/1763612757 2025-12-04T09:20:54.7317040Z * [new tag] viable/strict/1763616325 -> viable/strict/1763616325 2025-12-04T09:20:54.7318232Z * [new tag] viable/strict/1763623509 -> viable/strict/1763623509 2025-12-04T09:20:54.7318370Z * [new tag] viable/strict/1763624984 -> viable/strict/1763624984 2025-12-04T09:20:54.7318584Z * [new tag] viable/strict/1763628796 -> viable/strict/1763628796 2025-12-04T09:20:54.7318736Z * [new tag] viable/strict/1763634343 -> viable/strict/1763634343 2025-12-04T09:20:54.7318920Z * [new tag] viable/strict/1763635867 -> viable/strict/1763635867 2025-12-04T09:20:54.7319076Z * [new tag] viable/strict/1763639382 -> viable/strict/1763639382 2025-12-04T09:20:54.7324049Z * [new tag] viable/strict/1763646626 -> viable/strict/1763646626 2025-12-04T09:20:54.7326073Z * [new tag] viable/strict/1763655997 -> viable/strict/1763655997 2025-12-04T09:20:54.7326254Z * [new tag] viable/strict/1763659444 -> viable/strict/1763659444 2025-12-04T09:20:54.7326385Z * [new tag] viable/strict/1763660992 -> viable/strict/1763660992 2025-12-04T09:20:54.7326512Z * [new tag] viable/strict/1763663201 -> viable/strict/1763663201 2025-12-04T09:20:54.7326656Z * [new tag] viable/strict/1763670362 -> viable/strict/1763670362 2025-12-04T09:20:54.7326780Z * [new tag] viable/strict/1763675378 -> viable/strict/1763675378 2025-12-04T09:20:54.7326910Z * [new tag] viable/strict/1763693343 -> viable/strict/1763693343 2025-12-04T09:20:54.7327035Z * [new tag] viable/strict/1763696088 -> viable/strict/1763696088 2025-12-04T09:20:54.7327157Z * [new tag] viable/strict/1763697343 -> viable/strict/1763697343 2025-12-04T09:20:54.7327296Z * [new tag] viable/strict/1763699165 -> viable/strict/1763699165 2025-12-04T09:20:54.7327426Z * [new tag] viable/strict/1763700660 -> viable/strict/1763700660 2025-12-04T09:20:54.7328260Z * [new tag] viable/strict/1763704209 -> viable/strict/1763704209 2025-12-04T09:20:54.7328702Z * [new tag] viable/strict/1763706411 -> viable/strict/1763706411 2025-12-04T09:20:54.7329597Z * [new tag] viable/strict/1763708082 -> viable/strict/1763708082 2025-12-04T09:20:54.7329768Z * [new tag] viable/strict/1763711381 -> viable/strict/1763711381 2025-12-04T09:20:54.7330892Z * [new tag] viable/strict/1763713593 -> viable/strict/1763713593 2025-12-04T09:20:54.7331743Z * [new tag] viable/strict/1763715201 -> viable/strict/1763715201 2025-12-04T09:20:54.7332141Z * [new tag] viable/strict/1763733017 -> viable/strict/1763733017 2025-12-04T09:20:54.7336105Z * [new tag] viable/strict/1763735108 -> viable/strict/1763735108 2025-12-04T09:20:54.7336410Z * [new tag] viable/strict/1763749579 -> viable/strict/1763749579 2025-12-04T09:20:54.7336756Z * [new tag] viable/strict/1763751113 -> viable/strict/1763751113 2025-12-04T09:20:54.7336883Z * [new tag] viable/strict/1763753035 -> viable/strict/1763753035 2025-12-04T09:20:54.7337106Z * [new tag] viable/strict/1763754578 -> viable/strict/1763754578 2025-12-04T09:20:54.7337581Z * [new tag] viable/strict/1763756748 -> viable/strict/1763756748 2025-12-04T09:20:54.7338243Z * [new tag] viable/strict/1763758205 -> viable/strict/1763758205 2025-12-04T09:20:54.7338841Z * [new tag] viable/strict/1763764050 -> viable/strict/1763764050 2025-12-04T09:20:54.7339046Z * [new tag] viable/strict/1763771887 -> viable/strict/1763771887 2025-12-04T09:20:54.7339180Z * [new tag] viable/strict/1763773920 -> viable/strict/1763773920 2025-12-04T09:20:54.7339603Z * [new tag] viable/strict/1763776501 -> viable/strict/1763776501 2025-12-04T09:20:54.7342007Z * [new tag] viable/strict/1763779437 -> viable/strict/1763779437 2025-12-04T09:20:54.7342173Z * [new tag] viable/strict/1763781038 -> viable/strict/1763781038 2025-12-04T09:20:54.7342463Z * [new tag] viable/strict/1763782245 -> viable/strict/1763782245 2025-12-04T09:20:54.7342608Z * [new tag] viable/strict/1763785568 -> viable/strict/1763785568 2025-12-04T09:20:54.7344201Z * [new tag] viable/strict/1763787006 -> viable/strict/1763787006 2025-12-04T09:20:54.7344390Z * [new tag] viable/strict/1763789103 -> viable/strict/1763789103 2025-12-04T09:20:54.7346498Z * [new tag] viable/strict/1763790578 -> viable/strict/1763790578 2025-12-04T09:20:54.7351720Z * [new tag] viable/strict/1763796275 -> viable/strict/1763796275 2025-12-04T09:20:54.7356451Z * [new tag] viable/strict/1763801465 -> viable/strict/1763801465 2025-12-04T09:20:54.7361146Z * [new tag] viable/strict/1763803522 -> viable/strict/1763803522 2025-12-04T09:20:54.7365210Z * [new tag] viable/strict/1763808581 -> viable/strict/1763808581 2025-12-04T09:20:54.7366886Z * [new tag] viable/strict/1763840977 -> viable/strict/1763840977 2025-12-04T09:20:54.7367007Z * [new tag] viable/strict/1763846659 -> viable/strict/1763846659 2025-12-04T09:20:54.7367124Z * [new tag] viable/strict/1763872065 -> viable/strict/1763872065 2025-12-04T09:20:54.7367447Z * [new tag] viable/strict/1763873648 -> viable/strict/1763873648 2025-12-04T09:20:54.7367598Z * [new tag] viable/strict/1763875506 -> viable/strict/1763875506 2025-12-04T09:20:54.7367723Z * [new tag] viable/strict/1763889904 -> viable/strict/1763889904 2025-12-04T09:20:54.7367848Z * [new tag] viable/strict/1763930999 -> viable/strict/1763930999 2025-12-04T09:20:54.7367961Z * [new tag] viable/strict/1763944964 -> viable/strict/1763944964 2025-12-04T09:20:54.7368079Z * [new tag] viable/strict/1763958474 -> viable/strict/1763958474 2025-12-04T09:20:54.7368250Z * [new tag] viable/strict/1763967263 -> viable/strict/1763967263 2025-12-04T09:20:54.7368376Z * [new tag] viable/strict/1763972803 -> viable/strict/1763972803 2025-12-04T09:20:54.7368488Z * [new tag] viable/strict/1763976376 -> viable/strict/1763976376 2025-12-04T09:20:54.7368601Z * [new tag] viable/strict/1763989404 -> viable/strict/1763989404 2025-12-04T09:20:54.7368722Z * [new tag] viable/strict/1763990887 -> viable/strict/1763990887 2025-12-04T09:20:54.7368832Z * [new tag] viable/strict/1764019919 -> viable/strict/1764019919 2025-12-04T09:20:54.7368965Z * [new tag] viable/strict/1764023134 -> viable/strict/1764023134 2025-12-04T09:20:54.7369077Z * [new tag] viable/strict/1764024593 -> viable/strict/1764024593 2025-12-04T09:20:54.7369188Z * [new tag] viable/strict/1764026706 -> viable/strict/1764026706 2025-12-04T09:20:54.7369311Z * [new tag] viable/strict/1764031139 -> viable/strict/1764031139 2025-12-04T09:20:54.7369422Z * [new tag] viable/strict/1764033131 -> viable/strict/1764033131 2025-12-04T09:20:54.7369534Z * [new tag] viable/strict/1764035725 -> viable/strict/1764035725 2025-12-04T09:20:54.7369653Z * [new tag] viable/strict/1764624265 -> viable/strict/1764624265 2025-12-04T09:20:54.7369764Z * [new tag] viable/strict/1764631514 -> viable/strict/1764631514 2025-12-04T09:20:54.7369886Z * [new tag] viable/strict/1764632987 -> viable/strict/1764632987 2025-12-04T09:20:54.7370133Z * [new tag] viable/strict/1764636063 -> viable/strict/1764636063 2025-12-04T09:20:54.7370248Z * [new tag] viable/strict/1764643975 -> viable/strict/1764643975 2025-12-04T09:20:54.7370367Z * [new tag] viable/strict/1764646859 -> viable/strict/1764646859 2025-12-04T09:20:54.7370527Z * [new tag] viable/strict/1764653120 -> viable/strict/1764653120 2025-12-04T09:20:54.7370650Z * [new tag] viable/strict/1764654632 -> viable/strict/1764654632 2025-12-04T09:20:54.7370765Z * [new tag] viable/strict/1764656821 -> viable/strict/1764656821 2025-12-04T09:20:54.7370880Z * [new tag] viable/strict/1764658557 -> viable/strict/1764658557 2025-12-04T09:20:54.7371003Z * [new tag] viable/strict/1764660333 -> viable/strict/1764660333 2025-12-04T09:20:54.7371116Z * [new tag] viable/strict/1764661812 -> viable/strict/1764661812 2025-12-04T09:20:54.7371231Z * [new tag] viable/strict/1764664023 -> viable/strict/1764664023 2025-12-04T09:20:54.7371365Z * [new tag] viable/strict/1764669150 -> viable/strict/1764669150 2025-12-04T09:20:54.7371658Z * [new tag] viable/strict/1764680709 -> viable/strict/1764680709 2025-12-04T09:20:54.7371804Z * [new tag] viable/strict/1764687619 -> viable/strict/1764687619 2025-12-04T09:20:54.7371931Z * [new tag] viable/strict/1764696355 -> viable/strict/1764696355 2025-12-04T09:20:54.7372053Z * [new tag] viable/strict/1764701767 -> viable/strict/1764701767 2025-12-04T09:20:54.7372182Z * [new tag] viable/strict/1764710768 -> viable/strict/1764710768 2025-12-04T09:20:54.7372607Z * [new tag] viable/strict/1764716202 -> viable/strict/1764716202 2025-12-04T09:20:54.7372757Z * [new tag] viable/strict/1764793566 -> viable/strict/1764793566 2025-12-04T09:20:54.7372888Z * [new tag] viable/strict/1764797093 -> viable/strict/1764797093 2025-12-04T09:20:54.7373076Z * [new tag] viable/strict/1764800729 -> viable/strict/1764800729 2025-12-04T09:20:54.7375632Z * [new tag] whc_flight_1 -> whc_flight_1 2025-12-04T09:20:54.7375775Z * [new tag] whc_flight_2 -> whc_flight_2 2025-12-04T09:20:54.7376091Z * [new tag] whc_flight_4 -> whc_flight_4 2025-12-04T09:20:54.7876754Z [command]/usr/bin/git rev-parse --verify --quiet ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32^{object} 2025-12-04T09:20:54.7909023Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:20:54.7910279Z ##[endgroup] 2025-12-04T09:20:54.7910485Z ##[group]Determining the checkout info 2025-12-04T09:20:54.7911993Z ##[endgroup] 2025-12-04T09:20:54.7916574Z [command]/usr/bin/git sparse-checkout disable 2025-12-04T09:20:54.7952351Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-12-04T09:20:54.7976966Z ##[group]Checking out the ref 2025-12-04T09:20:54.7985252Z [command]/usr/bin/git checkout --progress --force ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:20:55.8192525Z Updating files: 96% (19479/20121) 2025-12-04T09:20:55.8315796Z Updating files: 97% (19518/20121) 2025-12-04T09:20:55.8551087Z Updating files: 98% (19719/20121) 2025-12-04T09:20:55.8689539Z Updating files: 99% (19920/20121) 2025-12-04T09:20:55.8689846Z Updating files: 100% (20121/20121) 2025-12-04T09:20:55.8690064Z Updating files: 100% (20121/20121), done. 2025-12-04T09:20:55.8912029Z Note: switching to 'ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32'. 2025-12-04T09:20:55.8913679Z 2025-12-04T09:20:55.8914199Z You are in 'detached HEAD' state. You can look around, make experimental 2025-12-04T09:20:55.8914740Z changes and commit them, and you can discard any commits you make in this 2025-12-04T09:20:55.8918340Z state without impacting any branches by switching back to a branch. 2025-12-04T09:20:55.8918570Z 2025-12-04T09:20:55.8918734Z If you want to create a new branch to retain commits you create, you may 2025-12-04T09:20:55.8919252Z do so (now or later) by using -c with the switch command. Example: 2025-12-04T09:20:55.8919947Z 2025-12-04T09:20:55.8920197Z git switch -c 2025-12-04T09:20:55.8920610Z 2025-12-04T09:20:55.8920851Z Or undo this operation with: 2025-12-04T09:20:55.8921060Z 2025-12-04T09:20:55.8921271Z git switch - 2025-12-04T09:20:55.8921369Z 2025-12-04T09:20:55.8921640Z Turn off this advice by setting config variable advice.detachedHead to false 2025-12-04T09:20:55.8921975Z 2025-12-04T09:20:55.8922340Z HEAD is now at ffd9b0fb435 Resolve collective autotuning test failure on arm (#168919) 2025-12-04T09:20:55.8978596Z ##[endgroup] 2025-12-04T09:20:55.8978940Z ##[group]Setting up auth for fetching submodules 2025-12-04T09:20:55.8979351Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T09:20:55.9046744Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-12-04T09:20:55.9078653Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-12-04T09:20:55.9109357Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-12-04T09:20:55.9133319Z ##[endgroup] 2025-12-04T09:20:55.9133873Z ##[group]Fetching submodules 2025-12-04T09:20:55.9136533Z [command]/usr/bin/git submodule sync --recursive 2025-12-04T09:20:55.9448023Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-12-04T09:20:55.9762033Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-12-04T09:20:55.9762665Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-12-04T09:20:55.9763257Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-12-04T09:20:55.9763805Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-12-04T09:20:55.9764310Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-12-04T09:20:55.9764969Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-12-04T09:20:55.9765650Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-12-04T09:20:55.9766133Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-12-04T09:20:55.9767347Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-12-04T09:20:55.9769209Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-12-04T09:20:55.9770892Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-12-04T09:20:55.9777002Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-12-04T09:20:55.9781734Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-12-04T09:20:55.9783703Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-12-04T09:20:55.9784391Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-12-04T09:20:55.9789936Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-12-04T09:20:55.9791986Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-12-04T09:20:55.9792723Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-12-04T09:20:55.9793624Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:20:55.9795473Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-12-04T09:20:55.9796184Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-12-04T09:20:55.9800288Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-12-04T09:20:55.9800965Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-12-04T09:20:55.9801483Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-12-04T09:20:55.9802029Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-12-04T09:20:55.9805233Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-12-04T09:20:55.9805895Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-12-04T09:20:55.9808143Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-12-04T09:20:55.9810683Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-12-04T09:20:55.9813300Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-12-04T09:20:55.9816519Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-12-04T09:20:55.9821269Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-12-04T09:20:55.9822058Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-12-04T09:20:55.9826668Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-12-04T09:20:55.9833321Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-12-04T09:20:55.9834981Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-12-04T09:20:55.9835592Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-12-04T09:20:55.9865810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-12-04T09:20:56.2104252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-12-04T09:20:56.2104806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-12-04T09:20:56.2105247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-12-04T09:20:56.2105686Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-12-04T09:20:56.2130603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-12-04T09:20:56.5223688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-12-04T09:20:56.5224889Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-12-04T09:20:56.5225740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-12-04T09:20:56.5226696Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-12-04T09:20:56.5227721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-12-04T09:20:56.5228573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-12-04T09:20:56.5229393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-12-04T09:20:56.5721868Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-12-04T09:20:57.3732546Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-12-04T09:20:57.3733845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-12-04T09:20:57.3734891Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-12-04T09:20:57.3736200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-12-04T09:20:57.3737122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-12-04T09:20:57.3737983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-12-04T09:20:57.3738817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-12-04T09:20:57.3739628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-12-04T09:20:57.3740441Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-12-04T09:20:57.4734054Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-12-04T09:20:57.7283673Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-12-04T09:20:57.7284810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-12-04T09:20:57.7838794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-12-04T09:21:12.0143352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-12-04T09:21:12.0144044Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-12-04T09:21:12.0144488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-12-04T09:21:12.0144958Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-12-04T09:21:12.0145350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-12-04T09:21:12.0145775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-12-04T09:21:12.0146200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-12-04T09:21:12.0146671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-12-04T09:21:12.0147096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-12-04T09:21:12.0147488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-12-04T09:21:12.0284350Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-12-04T09:21:12.0403563Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-12-04T09:21:12.0497292Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-12-04T09:21:12.0706230Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-12-04T09:21:12.1446087Z Submodule path 'third_party/NVTX': checked out '3ebbc93ded7285963bff932c678fa367eb393ba6' 2025-12-04T09:21:12.1892539Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-12-04T09:21:12.7120409Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-12-04T09:21:12.8703503Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-12-04T09:21:12.8720928Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:21:12.8753336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-12-04T09:21:17.8915732Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-12-04T09:21:17.9116117Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-12-04T09:21:18.1868462Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T09:21:18.2287488Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-12-04T09:21:18.3133866Z Submodule path 'third_party/cpuinfo': checked out 'f858c30bcb16f8effd5ff46996f0514539e17abc' 2025-12-04T09:21:18.3570663Z Submodule path 'third_party/cudnn_frontend': checked out '0b1577c8c83401237d601d0d0db5210506705396' 2025-12-04T09:21:18.8876872Z Submodule path 'third_party/cutlass': checked out 'f88806b1e31dfa579842638740216dd41fc6c588' 2025-12-04T09:21:19.0072050Z Submodule path 'third_party/fbgemm': checked out 'c0b988d39a9e47c794d699f29930ed4d7c7e13a4' 2025-12-04T09:21:19.0089037Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-12-04T09:21:19.0089729Z Submodule 'external/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:21:19.0090397Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:21:19.0090972Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-12-04T09:21:19.0095773Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-12-04T09:21:19.0096487Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:21:19.0097104Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-12-04T09:21:19.0126843Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-12-04T09:21:20.2447682Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-12-04T09:21:20.2449063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-12-04T09:21:20.2449734Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-12-04T09:21:20.3447940Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-12-04T09:21:23.8606949Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-12-04T09:21:23.9611838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-12-04T09:21:25.9251668Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-12-04T09:21:26.1957745Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T09:21:26.2838740Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-12-04T09:21:26.8067773Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '98125ce499b0fdf7ffbe0e3052f5b8709f4840f8' 2025-12-04T09:21:26.8471729Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:21:26.8594600Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-12-04T09:21:26.9461946Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-12-04T09:21:27.0029556Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-12-04T09:21:27.0043876Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:21:27.0046143Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:21:27.0071096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-12-04T09:21:31.7655017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-12-04T09:21:31.9522355Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-12-04T09:21:32.4117828Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-12-04T09:21:32.5191211Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-12-04T09:21:32.5449303Z Submodule path 'third_party/fmt': checked out '407c905e45ad75fc29bf0f9bb7c5c2fd3475976f' 2025-12-04T09:21:32.5796012Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-12-04T09:21:32.6010096Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341' 2025-12-04T09:21:32.6397395Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:21:32.6521628Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-12-04T09:21:32.6533298Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-12-04T09:21:32.6561700Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-12-04T09:21:48.8214971Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-12-04T09:21:48.8385888Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-12-04T09:21:48.9184924Z Submodule path 'third_party/kineto': checked out '31f85df8fbd89c188f14ef10f1ec65379786b943' 2025-12-04T09:21:48.9199635Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:21:48.9200831Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:21:48.9201679Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:21:48.9223729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-12-04T09:21:50.1210534Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-12-04T09:21:50.3500579Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-12-04T09:21:50.4245468Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1' 2025-12-04T09:21:50.4264019Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:21:50.4266613Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:21:50.4268223Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:21:50.4268887Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:21:50.4269602Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:21:50.4270310Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:21:50.4271007Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:21:50.4272467Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:21:50.4273309Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:21:50.4298779Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-12-04T09:21:52.1736797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-12-04T09:21:52.1738117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'... 2025-12-04T09:21:52.1738871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-12-04T09:21:52.1739578Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-12-04T09:21:52.1740267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-12-04T09:21:52.1741005Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-12-04T09:21:52.1741723Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-12-04T09:21:52.2739825Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-12-04T09:21:57.2482449Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-12-04T09:21:57.2644430Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-12-04T09:21:57.2958694Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-12-04T09:21:57.3076964Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-12-04T09:21:57.3092284Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:21:57.3118592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-12-04T09:21:57.9350008Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-12-04T09:21:57.9507664Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-12-04T09:21:57.9887566Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:21:58.0719461Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-12-04T09:21:58.0860655Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-12-04T09:21:58.1005753Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a' 2025-12-04T09:21:58.1019293Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:21:58.1021774Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:21:58.1048800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T09:21:59.9768873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T09:22:00.1846861Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159' 2025-12-04T09:22:00.2234444Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T09:22:00.2516007Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-12-04T09:22:00.2896029Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T09:22:00.3330667Z Submodule path 'third_party/kleidiai': checked out 'd7770c89632329a9914ef1a90289917597639cbe' 2025-12-04T09:22:00.3662240Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-12-04T09:22:00.4522592Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-12-04T09:22:00.7298281Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-12-04T09:22:00.7330531Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-12-04T09:22:00.7358882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-12-04T09:22:01.6355195Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-12-04T09:22:01.6902688Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-12-04T09:22:01.6922123Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:22:01.6928819Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:22:01.6930512Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:22:01.6931190Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:22:01.6932134Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:22:01.6940920Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:22:01.6945967Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:22:01.6950420Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:22:01.6962827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-12-04T09:22:02.1038840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-12-04T09:22:02.1039597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-12-04T09:22:02.1040258Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-12-04T09:22:02.1041114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-12-04T09:22:02.2046111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-12-04T09:22:02.7254893Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-12-04T09:22:08.5392088Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-12-04T09:22:09.2737400Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-12-04T09:22:09.3087857Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-12-04T09:22:09.3246828Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-12-04T09:22:09.4155643Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-12-04T09:22:09.4278795Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-12-04T09:22:09.4401028Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-12-04T09:22:09.4531421Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-12-04T09:22:09.4546649Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:09.4547431Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:09.4573647Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T09:22:11.2612861Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T09:22:11.4690547Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-12-04T09:22:11.5079564Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T09:22:11.8255298Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-12-04T09:22:11.8363506Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-12-04T09:22:12.0524852Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-12-04T09:22:12.0543987Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:22:12.0551412Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-12-04T09:22:12.0572917Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-12-04T09:22:12.5590111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-12-04T09:22:12.9537891Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-12-04T09:22:13.0153404Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-12-04T09:22:13.0242315Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-12-04T09:22:13.0358207Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-12-04T09:22:13.0728936Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-12-04T09:22:13.0977871Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-12-04T09:22:13.1349297Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-12-04T09:22:13.1571132Z Submodule path 'third_party/tensorpipe': checked out '2b4cd91092d335a697416b2a3cb398283246849d' 2025-12-04T09:22:13.1592094Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:22:13.1592838Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:22:13.1593500Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:22:13.1594137Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:22:13.1617421Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-12-04T09:22:14.0751941Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-12-04T09:22:14.0752570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-12-04T09:22:14.1677188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-12-04T09:22:14.2181340Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-12-04T09:22:14.2308630Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-12-04T09:22:14.2926337Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-12-04T09:22:14.3182338Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-12-04T09:22:14.3192711Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:22:14.3218664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-12-04T09:22:14.4983919Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-12-04T09:22:14.5033808Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-12-04T09:22:14.5350663Z Entering 'android/libs/fbjni' 2025-12-04T09:22:14.5388601Z Entering 'third_party/FP16' 2025-12-04T09:22:14.5435538Z Entering 'third_party/FXdiv' 2025-12-04T09:22:14.5474793Z Entering 'third_party/NNPACK' 2025-12-04T09:22:14.5512238Z Entering 'third_party/NVTX' 2025-12-04T09:22:14.5557371Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:22:14.5599877Z Entering 'third_party/XNNPACK' 2025-12-04T09:22:14.5651673Z Entering 'third_party/aiter' 2025-12-04T09:22:14.5692541Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:22:14.5740345Z Entering 'third_party/benchmark' 2025-12-04T09:22:14.5780365Z Entering 'third_party/composable_kernel' 2025-12-04T09:22:14.5831849Z Entering 'third_party/cpp-httplib' 2025-12-04T09:22:14.5866701Z Entering 'third_party/cpuinfo' 2025-12-04T09:22:14.5907732Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:22:14.5954007Z Entering 'third_party/cutlass' 2025-12-04T09:22:14.5997772Z Entering 'third_party/fbgemm' 2025-12-04T09:22:14.6038002Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:22:14.6073163Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:22:14.6115630Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:22:14.6155075Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:22:14.6206209Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:22:14.6248028Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:22:14.6286934Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:22:14.6327534Z Entering 'third_party/flash-attention' 2025-12-04T09:22:14.6366930Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:22:14.6411767Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:22:14.6463547Z Entering 'third_party/flatbuffers' 2025-12-04T09:22:14.6506524Z Entering 'third_party/fmt' 2025-12-04T09:22:14.6551618Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:22:14.6583657Z Entering 'third_party/gloo' 2025-12-04T09:22:14.6623117Z Entering 'third_party/googletest' 2025-12-04T09:22:14.6663573Z Entering 'third_party/ideep' 2025-12-04T09:22:14.6704428Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:22:14.6757748Z Entering 'third_party/ittapi' 2025-12-04T09:22:14.6799930Z Entering 'third_party/kineto' 2025-12-04T09:22:14.6835140Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:22:14.6868993Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:22:14.6913920Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:22:14.6950349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:22:14.6991139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:22:14.7031050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:22:14.7079536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:22:14.7114500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:22:14.7154856Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:22:14.7197164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:22:14.7236365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:22:14.7276728Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:14.7314383Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:14.7353358Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:22:14.7392377Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:22:14.7433077Z Entering 'third_party/kleidiai' 2025-12-04T09:22:14.7470193Z Entering 'third_party/mimalloc' 2025-12-04T09:22:14.7512710Z Entering 'third_party/nlohmann' 2025-12-04T09:22:14.7555681Z Entering 'third_party/onnx' 2025-12-04T09:22:14.7604451Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:22:14.7655211Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:22:14.7694256Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:22:14.7730907Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:22:14.7769378Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:22:14.7809257Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:22:14.7846422Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:22:14.7886875Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:22:14.7928706Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:22:14.7962563Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:14.8007695Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:14.8048316Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:22:14.8106818Z Entering 'third_party/pocketfft' 2025-12-04T09:22:14.8146549Z Entering 'third_party/protobuf' 2025-12-04T09:22:14.8186470Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:22:14.8222884Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:22:14.8264945Z Entering 'third_party/psimd' 2025-12-04T09:22:14.8308620Z Entering 'third_party/pthreadpool' 2025-12-04T09:22:14.8348567Z Entering 'third_party/pybind11' 2025-12-04T09:22:14.8390067Z Entering 'third_party/python-peachpy' 2025-12-04T09:22:14.8429011Z Entering 'third_party/sleef' 2025-12-04T09:22:14.8465306Z Entering 'third_party/tensorpipe' 2025-12-04T09:22:14.8508220Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:22:14.8548407Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:22:14.8581974Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:22:14.8618716Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:22:14.8653300Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:22:14.8714433Z ##[endgroup] 2025-12-04T09:22:14.8714891Z ##[group]Persisting credentials for submodules 2025-12-04T09:22:14.8721103Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-12-04T09:22:14.9025923Z Entering 'android/libs/fbjni' 2025-12-04T09:22:14.9090285Z Entering 'third_party/FP16' 2025-12-04T09:22:14.9144916Z Entering 'third_party/FXdiv' 2025-12-04T09:22:14.9193394Z Entering 'third_party/NNPACK' 2025-12-04T09:22:14.9249844Z Entering 'third_party/NVTX' 2025-12-04T09:22:14.9305211Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:22:14.9356653Z Entering 'third_party/XNNPACK' 2025-12-04T09:22:14.9421588Z Entering 'third_party/aiter' 2025-12-04T09:22:14.9478411Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:22:14.9545928Z Entering 'third_party/benchmark' 2025-12-04T09:22:14.9602900Z Entering 'third_party/composable_kernel' 2025-12-04T09:22:14.9664584Z Entering 'third_party/cpp-httplib' 2025-12-04T09:22:14.9725022Z Entering 'third_party/cpuinfo' 2025-12-04T09:22:14.9779771Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:22:14.9835571Z Entering 'third_party/cutlass' 2025-12-04T09:22:14.9896629Z Entering 'third_party/fbgemm' 2025-12-04T09:22:14.9952060Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:22:15.0006576Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:22:15.0057138Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:22:15.0110366Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:22:15.0169977Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:22:15.0223871Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:22:15.0276297Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:22:15.0333650Z Entering 'third_party/flash-attention' 2025-12-04T09:22:15.0386481Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:22:15.0442718Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:22:15.0509645Z Entering 'third_party/flatbuffers' 2025-12-04T09:22:15.0563824Z Entering 'third_party/fmt' 2025-12-04T09:22:15.0615995Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:22:15.0669428Z Entering 'third_party/gloo' 2025-12-04T09:22:15.0725835Z Entering 'third_party/googletest' 2025-12-04T09:22:15.0782542Z Entering 'third_party/ideep' 2025-12-04T09:22:15.0834857Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:22:15.0898606Z Entering 'third_party/ittapi' 2025-12-04T09:22:15.0957376Z Entering 'third_party/kineto' 2025-12-04T09:22:15.1011975Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:22:15.1063460Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:22:15.1121494Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:22:15.1167139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:22:15.1217284Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:22:15.1268168Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:22:15.1326644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:22:15.1381494Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:22:15.1432320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:22:15.1486671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:22:15.1537917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:22:15.1590163Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:15.1646494Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:15.1711201Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:22:15.1764394Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:22:15.1823538Z Entering 'third_party/kleidiai' 2025-12-04T09:22:15.1877392Z Entering 'third_party/mimalloc' 2025-12-04T09:22:15.1931131Z Entering 'third_party/nlohmann' 2025-12-04T09:22:15.1987264Z Entering 'third_party/onnx' 2025-12-04T09:22:15.2050714Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:22:15.2102807Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:22:15.2156912Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:22:15.2206429Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:22:15.2259707Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:22:15.2316300Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:22:15.2370765Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:22:15.2420437Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:22:15.2470827Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:22:15.2524980Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:15.2571167Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:15.2627019Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:22:15.2691114Z Entering 'third_party/pocketfft' 2025-12-04T09:22:15.2744265Z Entering 'third_party/protobuf' 2025-12-04T09:22:15.2797521Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:22:15.2848704Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:22:15.2909792Z Entering 'third_party/psimd' 2025-12-04T09:22:15.2963638Z Entering 'third_party/pthreadpool' 2025-12-04T09:22:15.3018477Z Entering 'third_party/pybind11' 2025-12-04T09:22:15.3071779Z Entering 'third_party/python-peachpy' 2025-12-04T09:22:15.3126436Z Entering 'third_party/sleef' 2025-12-04T09:22:15.3185523Z Entering 'third_party/tensorpipe' 2025-12-04T09:22:15.3236815Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:22:15.3287890Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:22:15.3344113Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:22:15.3399963Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:22:15.3452502Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:22:15.3525046Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-12-04T09:22:15.3833759Z Entering 'android/libs/fbjni' 2025-12-04T09:22:15.3880616Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T09:22:15.3893668Z Entering 'third_party/FP16' 2025-12-04T09:22:15.3939843Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T09:22:15.3956717Z Entering 'third_party/FXdiv' 2025-12-04T09:22:15.4001556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T09:22:15.4015334Z Entering 'third_party/NNPACK' 2025-12-04T09:22:15.4064181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T09:22:15.4077666Z Entering 'third_party/NVTX' 2025-12-04T09:22:15.4120179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T09:22:15.4140701Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:22:15.4185325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T09:22:15.4200249Z Entering 'third_party/XNNPACK' 2025-12-04T09:22:15.4250084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T09:22:15.4277218Z Entering 'third_party/aiter' 2025-12-04T09:22:15.4324660Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T09:22:15.4339352Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:22:15.4384068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T09:22:15.4408057Z Entering 'third_party/benchmark' 2025-12-04T09:22:15.4454286Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:22:15.4471256Z Entering 'third_party/composable_kernel' 2025-12-04T09:22:15.4524430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T09:22:15.4545974Z Entering 'third_party/cpp-httplib' 2025-12-04T09:22:15.4593055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T09:22:15.4608613Z Entering 'third_party/cpuinfo' 2025-12-04T09:22:15.4655906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T09:22:15.4671026Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:22:15.4723793Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T09:22:15.4740743Z Entering 'third_party/cutlass' 2025-12-04T09:22:15.4786081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T09:22:15.4808468Z Entering 'third_party/fbgemm' 2025-12-04T09:22:15.4861819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T09:22:15.4882845Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:22:15.4926902Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T09:22:15.4946858Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:22:15.4991194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T09:22:15.5015715Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:22:15.5060835Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T09:22:15.5072943Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:22:15.5121406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T09:22:15.5144497Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:22:15.5192922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T09:22:15.5205723Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:22:15.5255765Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T09:22:15.5265022Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:22:15.5312176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T09:22:15.5332583Z Entering 'third_party/flash-attention' 2025-12-04T09:22:15.5385018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T09:22:15.5400638Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:22:15.5442874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T09:22:15.5462323Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:22:15.5510326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T09:22:15.5537506Z Entering 'third_party/flatbuffers' 2025-12-04T09:22:15.5584335Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T09:22:15.5602553Z Entering 'third_party/fmt' 2025-12-04T09:22:15.5646817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:22:15.5668243Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:22:15.5720747Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T09:22:15.5733512Z Entering 'third_party/gloo' 2025-12-04T09:22:15.5777331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T09:22:15.5795292Z Entering 'third_party/googletest' 2025-12-04T09:22:15.5840142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:22:15.5857740Z Entering 'third_party/ideep' 2025-12-04T09:22:15.5907383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T09:22:15.5919327Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:22:15.5964218Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T09:22:15.5993268Z Entering 'third_party/ittapi' 2025-12-04T09:22:15.6034267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T09:22:15.6048368Z Entering 'third_party/kineto' 2025-12-04T09:22:15.6099399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T09:22:15.6116900Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:22:15.6157681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T09:22:15.6176919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:22:15.6225722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T09:22:15.6246059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:22:15.6293407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T09:22:15.6308327Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:22:15.6358883Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:22:15.6369871Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:22:15.6418728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T09:22:15.6436043Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:22:15.6478071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T09:22:15.6500760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:22:15.6549407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T09:22:15.6567293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:22:15.6615443Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:22:15.6636544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:22:15.6683368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T09:22:15.6700310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:22:15.6746343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T09:22:15.6763889Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:22:15.6811284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:22:15.6830769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:15.6877324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:22:15.6897133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:15.6943830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:22:15.6968865Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:22:15.7010000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T09:22:15.7025671Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:22:15.7072259Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T09:22:15.7096376Z Entering 'third_party/kleidiai' 2025-12-04T09:22:15.7142533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T09:22:15.7157584Z Entering 'third_party/mimalloc' 2025-12-04T09:22:15.7205664Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T09:22:15.7224890Z Entering 'third_party/nlohmann' 2025-12-04T09:22:15.7274195Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T09:22:15.7295611Z Entering 'third_party/onnx' 2025-12-04T09:22:15.7342256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T09:22:15.7372158Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:22:15.7420270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:22:15.7444741Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:22:15.7494105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T09:22:15.7508793Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:22:15.7555944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:22:15.7570526Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:22:15.7623968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:22:15.7635427Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:22:15.7685081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T09:22:15.7698830Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:22:15.7750043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T09:22:15.7766035Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:22:15.7814251Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T09:22:15.7831404Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:22:15.7880428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T09:22:15.7893852Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:22:15.7940400Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:22:15.7956991Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:15.8004819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:22:15.8024652Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:15.8065634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:22:15.8085853Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:22:15.8133004Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T09:22:15.8165378Z Entering 'third_party/pocketfft' 2025-12-04T09:22:15.8213877Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T09:22:15.8232503Z Entering 'third_party/protobuf' 2025-12-04T09:22:15.8275599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T09:22:15.8297164Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:22:15.8348711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:22:15.8361892Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:22:15.8405901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:22:15.8422873Z Entering 'third_party/psimd' 2025-12-04T09:22:15.8477022Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T09:22:15.8492349Z Entering 'third_party/pthreadpool' 2025-12-04T09:22:15.8543460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T09:22:15.8560485Z Entering 'third_party/pybind11' 2025-12-04T09:22:15.8607581Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:22:15.8628499Z Entering 'third_party/python-peachpy' 2025-12-04T09:22:15.8665756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T09:22:15.8686033Z Entering 'third_party/sleef' 2025-12-04T09:22:15.8729649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T09:22:15.8749039Z Entering 'third_party/tensorpipe' 2025-12-04T09:22:15.8796260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T09:22:15.8808185Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:22:15.8854010Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:22:15.8870007Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:22:15.8919663Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T09:22:15.8939331Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:22:15.8980586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T09:22:15.9001117Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:22:15.9047556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:22:15.9059815Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:22:15.9108167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T09:22:16.0100517Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-12-04T09:22:16.0426010Z Entering 'android/libs/fbjni' 2025-12-04T09:22:16.0464094Z Entering 'third_party/FP16' 2025-12-04T09:22:16.0507773Z Entering 'third_party/FXdiv' 2025-12-04T09:22:16.0544682Z Entering 'third_party/NNPACK' 2025-12-04T09:22:16.0584917Z Entering 'third_party/NVTX' 2025-12-04T09:22:16.0628454Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:22:16.0665232Z Entering 'third_party/XNNPACK' 2025-12-04T09:22:16.0720203Z Entering 'third_party/aiter' 2025-12-04T09:22:16.0757665Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:22:16.0805483Z Entering 'third_party/benchmark' 2025-12-04T09:22:16.0847109Z Entering 'third_party/composable_kernel' 2025-12-04T09:22:16.0890851Z Entering 'third_party/cpp-httplib' 2025-12-04T09:22:16.0929500Z Entering 'third_party/cpuinfo' 2025-12-04T09:22:16.0973650Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:22:16.1013166Z Entering 'third_party/cutlass' 2025-12-04T09:22:16.1060199Z Entering 'third_party/fbgemm' 2025-12-04T09:22:16.1100086Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:22:16.1149875Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:22:16.1192547Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:22:16.1235259Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:22:16.1276822Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:22:16.1314259Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:22:16.1350925Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:22:16.1393260Z Entering 'third_party/flash-attention' 2025-12-04T09:22:16.1431674Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:22:16.1474147Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:22:16.1522945Z Entering 'third_party/flatbuffers' 2025-12-04T09:22:16.1566650Z Entering 'third_party/fmt' 2025-12-04T09:22:16.1610808Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:22:16.1652405Z Entering 'third_party/gloo' 2025-12-04T09:22:16.1688186Z Entering 'third_party/googletest' 2025-12-04T09:22:16.1732093Z Entering 'third_party/ideep' 2025-12-04T09:22:16.1766293Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:22:16.1810368Z Entering 'third_party/ittapi' 2025-12-04T09:22:16.1854341Z Entering 'third_party/kineto' 2025-12-04T09:22:16.1897969Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:22:16.1934276Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:22:16.1973018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:22:16.2012526Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:22:16.2057237Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:22:16.2094132Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:22:16.2138465Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:22:16.2173237Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:22:16.2210901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:22:16.2252082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:22:16.2292542Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:22:16.2331934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:16.2369033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:16.2418158Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:22:16.2454985Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:22:16.2499984Z Entering 'third_party/kleidiai' 2025-12-04T09:22:16.2539459Z Entering 'third_party/mimalloc' 2025-12-04T09:22:16.2580063Z Entering 'third_party/nlohmann' 2025-12-04T09:22:16.2616841Z Entering 'third_party/onnx' 2025-12-04T09:22:16.2669403Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:22:16.2716026Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:22:16.2754314Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:22:16.2795155Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:22:16.2830051Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:22:16.2869850Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:22:16.2912549Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:22:16.2946917Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:22:16.2979323Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:22:16.3019415Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:16.3059680Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:16.3104423Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:22:16.3154899Z Entering 'third_party/pocketfft' 2025-12-04T09:22:16.3194510Z Entering 'third_party/protobuf' 2025-12-04T09:22:16.3243649Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:22:16.3283313Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:22:16.3327060Z Entering 'third_party/psimd' 2025-12-04T09:22:16.3365495Z Entering 'third_party/pthreadpool' 2025-12-04T09:22:16.3407466Z Entering 'third_party/pybind11' 2025-12-04T09:22:16.3448914Z Entering 'third_party/python-peachpy' 2025-12-04T09:22:16.3492452Z Entering 'third_party/sleef' 2025-12-04T09:22:16.3529077Z Entering 'third_party/tensorpipe' 2025-12-04T09:22:16.3573730Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:22:16.3612582Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:22:16.3653703Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:22:16.3701125Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:22:16.3737430Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:22:16.3795491Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-12-04T09:22:16.4106863Z Entering 'android/libs/fbjni' 2025-12-04T09:22:16.4146395Z Entering 'third_party/FP16' 2025-12-04T09:22:16.4183901Z Entering 'third_party/FXdiv' 2025-12-04T09:22:16.4225028Z Entering 'third_party/NNPACK' 2025-12-04T09:22:16.4260431Z Entering 'third_party/NVTX' 2025-12-04T09:22:16.4301852Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:22:16.4347028Z Entering 'third_party/XNNPACK' 2025-12-04T09:22:16.4401493Z Entering 'third_party/aiter' 2025-12-04T09:22:16.4439267Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:22:16.4491764Z Entering 'third_party/benchmark' 2025-12-04T09:22:16.4529023Z Entering 'third_party/composable_kernel' 2025-12-04T09:22:16.4572861Z Entering 'third_party/cpp-httplib' 2025-12-04T09:22:16.4611953Z Entering 'third_party/cpuinfo' 2025-12-04T09:22:16.4652134Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:22:16.4689221Z Entering 'third_party/cutlass' 2025-12-04T09:22:16.4736405Z Entering 'third_party/fbgemm' 2025-12-04T09:22:16.4777786Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:22:16.4814401Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:22:16.4861123Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:22:16.4900929Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:22:16.4952704Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:22:16.4989398Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:22:16.5029045Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:22:16.5070208Z Entering 'third_party/flash-attention' 2025-12-04T09:22:16.5110417Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:22:16.5150330Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:22:16.5195258Z Entering 'third_party/flatbuffers' 2025-12-04T09:22:16.5240436Z Entering 'third_party/fmt' 2025-12-04T09:22:16.5274120Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:22:16.5320502Z Entering 'third_party/gloo' 2025-12-04T09:22:16.5358942Z Entering 'third_party/googletest' 2025-12-04T09:22:16.5399193Z Entering 'third_party/ideep' 2025-12-04T09:22:16.5432272Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:22:16.5477743Z Entering 'third_party/ittapi' 2025-12-04T09:22:16.5512448Z Entering 'third_party/kineto' 2025-12-04T09:22:16.5551308Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:22:16.5587261Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:22:16.5625872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:22:16.5664443Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:22:16.5704351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:22:16.5743676Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:22:16.5788952Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:22:16.5828168Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:22:16.5864213Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:22:16.5904882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:22:16.5946105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:22:16.5983058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:16.6026279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:16.6070376Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:22:16.6113825Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:22:16.6155635Z Entering 'third_party/kleidiai' 2025-12-04T09:22:16.6197643Z Entering 'third_party/mimalloc' 2025-12-04T09:22:16.6237839Z Entering 'third_party/nlohmann' 2025-12-04T09:22:16.6276832Z Entering 'third_party/onnx' 2025-12-04T09:22:16.6332151Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:22:16.6376039Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:22:16.6417060Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:22:16.6449406Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:22:16.6488653Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:22:16.6529422Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:22:16.6567360Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:22:16.6604176Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:22:16.6644424Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:22:16.6681954Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:16.6719293Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:16.6757591Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:22:16.6811460Z Entering 'third_party/pocketfft' 2025-12-04T09:22:16.6852098Z Entering 'third_party/protobuf' 2025-12-04T09:22:16.6896521Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:22:16.6932044Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:22:16.6981411Z Entering 'third_party/psimd' 2025-12-04T09:22:16.7019541Z Entering 'third_party/pthreadpool' 2025-12-04T09:22:16.7062867Z Entering 'third_party/pybind11' 2025-12-04T09:22:16.7102796Z Entering 'third_party/python-peachpy' 2025-12-04T09:22:16.7143571Z Entering 'third_party/sleef' 2025-12-04T09:22:16.7181222Z Entering 'third_party/tensorpipe' 2025-12-04T09:22:16.7223795Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:22:16.7259441Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:22:16.7296796Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:22:16.7330943Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:22:16.7370337Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:22:16.7429070Z ##[endgroup] 2025-12-04T09:22:16.7466268Z [command]/usr/bin/git log -1 --format=%H 2025-12-04T09:22:16.7486604Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:22:16.7583161Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-12-04T09:22:16.7583424Z cd "${GITHUB_WORKSPACE}" 2025-12-04T09:22:16.7583615Z # Clean stale submodule dirs 2025-12-04T09:22:16.7583809Z if [ -z "${NO_SUDO}" ]; then 2025-12-04T09:22:16.7584039Z  sudo git submodule foreach --recursive git clean -ffdx 2025-12-04T09:22:16.7584258Z else 2025-12-04T09:22:16.7584450Z  git submodule foreach --recursive git clean -ffdx 2025-12-04T09:22:16.7584677Z fi 2025-12-04T09:22:16.7591737Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:16.7591977Z env: 2025-12-04T09:22:16.7592131Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:16.7592294Z NO_SUDO: true 2025-12-04T09:22:16.7592447Z ##[endgroup] 2025-12-04T09:22:16.7895892Z Entering 'android/libs/fbjni' 2025-12-04T09:22:16.7933150Z Entering 'third_party/FP16' 2025-12-04T09:22:16.7965734Z Entering 'third_party/FXdiv' 2025-12-04T09:22:16.7996094Z Entering 'third_party/NNPACK' 2025-12-04T09:22:16.8028450Z Entering 'third_party/NVTX' 2025-12-04T09:22:16.8057430Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:22:16.8088049Z Entering 'third_party/XNNPACK' 2025-12-04T09:22:16.8186213Z Entering 'third_party/aiter' 2025-12-04T09:22:16.8224555Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:22:16.8319443Z Entering 'third_party/benchmark' 2025-12-04T09:22:16.8351547Z Entering 'third_party/composable_kernel' 2025-12-04T09:22:16.8440820Z Entering 'third_party/cpp-httplib' 2025-12-04T09:22:16.8469103Z Entering 'third_party/cpuinfo' 2025-12-04T09:22:16.8502335Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:22:16.8536891Z Entering 'third_party/cutlass' 2025-12-04T09:22:16.8623369Z Entering 'third_party/fbgemm' 2025-12-04T09:22:16.8667800Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:22:16.8700681Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:22:16.8793518Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:22:16.8821795Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:22:16.8902465Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:22:16.8932881Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:22:16.8961001Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:22:16.9002660Z Entering 'third_party/flash-attention' 2025-12-04T09:22:16.9038436Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:22:16.9114905Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:22:16.9189867Z Entering 'third_party/flatbuffers' 2025-12-04T09:22:16.9250588Z Entering 'third_party/fmt' 2025-12-04T09:22:16.9286295Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:22:16.9320018Z Entering 'third_party/gloo' 2025-12-04T09:22:16.9355461Z Entering 'third_party/googletest' 2025-12-04T09:22:16.9386261Z Entering 'third_party/ideep' 2025-12-04T09:22:16.9411827Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:22:16.9482065Z Entering 'third_party/ittapi' 2025-12-04T09:22:16.9511983Z Entering 'third_party/kineto' 2025-12-04T09:22:16.9543890Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:22:16.9578353Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:22:16.9613043Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:22:16.9649290Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:22:16.9677805Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:22:16.9705472Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:22:16.9731868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:22:16.9760835Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:22:16.9792375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:22:16.9827256Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:22:16.9857122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:22:16.9882973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:16.9923050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:16.9956686Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:22:16.9988239Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:22:17.0024417Z Entering 'third_party/kleidiai' 2025-12-04T09:22:17.0061539Z Entering 'third_party/mimalloc' 2025-12-04T09:22:17.0096387Z Entering 'third_party/nlohmann' 2025-12-04T09:22:17.0136403Z Entering 'third_party/onnx' 2025-12-04T09:22:17.0367448Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:22:17.0403302Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:22:17.0447701Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:22:17.0481409Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:22:17.0511979Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:22:17.0539397Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:22:17.0583075Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:22:17.0610462Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:22:17.0642142Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:22:17.0670772Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:22:17.0713187Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:22:17.0744216Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:22:17.0925789Z Entering 'third_party/pocketfft' 2025-12-04T09:22:17.0956659Z Entering 'third_party/protobuf' 2025-12-04T09:22:17.1018409Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:22:17.1046378Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:22:17.1086170Z Entering 'third_party/psimd' 2025-12-04T09:22:17.1116815Z Entering 'third_party/pthreadpool' 2025-12-04T09:22:17.1146172Z Entering 'third_party/pybind11' 2025-12-04T09:22:17.1178481Z Entering 'third_party/python-peachpy' 2025-12-04T09:22:17.1206244Z Entering 'third_party/sleef' 2025-12-04T09:22:17.1240701Z Entering 'third_party/tensorpipe' 2025-12-04T09:22:17.1266456Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:22:17.1296216Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:22:17.1327716Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:22:17.1361002Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:22:17.1391970Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:22:17.1518221Z Prepare all required actions 2025-12-04T09:22:17.1518668Z Getting action download info 2025-12-04T09:22:17.3228368Z ##[group]Run ./.github/actions/setup-linux 2025-12-04T09:22:17.3228602Z env: 2025-12-04T09:22:17.3228772Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:17.3228961Z ##[endgroup] 2025-12-04T09:22:17.3261166Z ##[group]Run set -euo pipefail 2025-12-04T09:22:17.3261414Z set -euo pipefail 2025-12-04T09:22:17.3261606Z function get_ec2_metadata() { 2025-12-04T09:22:17.3261842Z  # Pulled from instance metadata endpoint for EC2 2025-12-04T09:22:17.3262208Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-12-04T09:22:17.3262533Z  category=$1 2025-12-04T09:22:17.3262762Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-12-04T09:22:17.3263015Z  runner_name_str=i-0cbeb234d1c75d5ac 2025-12-04T09:22:17.3263285Z  if [[ -f /.inarc ]]; then 2025-12-04T09:22:17.3263507Z  echo "ARC Runner, no info on ec2 metadata" 2025-12-04T09:22:17.3263743Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-12-04T09:22:17.3264019Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-12-04T09:22:17.3264266Z  else 2025-12-04T09:22:17.3264767Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-12-04T09:22:17.3265278Z  fi 2025-12-04T09:22:17.3265421Z } 2025-12-04T09:22:17.3265594Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-12-04T09:22:17.3265844Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-12-04T09:22:17.3266129Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-12-04T09:22:17.3266377Z echo "system info $(uname -a)" 2025-12-04T09:22:17.3271127Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:17.3271358Z env: 2025-12-04T09:22:17.3271511Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:17.3271674Z ##[endgroup] 2025-12-04T09:22:17.3403355Z ami-id: ami-08982f1c5bf93d976 2025-12-04T09:22:17.3509170Z instance-id: i-0cbeb234d1c75d5ac 2025-12-04T09:22:17.3609947Z instance-type: m7i-flex.8xlarge 2025-12-04T09:22:17.3621806Z system info Linux ip-10-0-31-90.ec2.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-12-04T09:22:17.3636868Z ##[group]Run if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T09:22:17.3637188Z if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T09:22:17.3641796Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:17.3642049Z env: 2025-12-04T09:22:17.3642203Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:17.3642381Z ##[endgroup] 2025-12-04T09:22:17.3676590Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:22:17.3677172Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:22:17.3681262Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:17.3681513Z env: 2025-12-04T09:22:17.3681672Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:17.3681856Z ##[endgroup] 2025-12-04T09:22:17.3726370Z ##[group]Run if systemctl is-active --quiet docker; then 2025-12-04T09:22:17.3726656Z if systemctl is-active --quiet docker; then 2025-12-04T09:22:17.3726902Z  echo "Docker daemon is running..."; 2025-12-04T09:22:17.3727112Z else 2025-12-04T09:22:17.3727339Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-12-04T09:22:17.3727599Z fi 2025-12-04T09:22:17.3731450Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:17.3731968Z env: 2025-12-04T09:22:17.3732122Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:17.3732311Z ##[endgroup] 2025-12-04T09:22:17.3804753Z Docker daemon is running... 2025-12-04T09:22:17.3834475Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T09:22:17.3834677Z with: 2025-12-04T09:22:17.3834818Z shell: bash 2025-12-04T09:22:17.3834961Z timeout_minutes: 5 2025-12-04T09:22:17.3835118Z max_attempts: 3 2025-12-04T09:22:17.3835271Z retry_wait_seconds: 30 2025-12-04T09:22:17.3836542Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-12-04T09:22:17.3837761Z polling_interval_seconds: 1 2025-12-04T09:22:17.3837929Z warning_on_retry: true 2025-12-04T09:22:17.3838092Z continue_on_error: false 2025-12-04T09:22:17.3838253Z env: 2025-12-04T09:22:17.3838381Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:17.3838550Z AWS_RETRY_MODE: standard 2025-12-04T09:22:17.3838713Z AWS_MAX_ATTEMPTS: 5 2025-12-04T09:22:17.3838875Z AWS_DEFAULT_REGION: us-east-1 2025-12-04T09:22:17.3839050Z ##[endgroup] 2025-12-04T09:22:18.3646350Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:22:18.3646818Z Configure a credential helper to remove this warning. See 2025-12-04T09:22:18.3647230Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:22:18.3647487Z 2025-12-04T09:22:18.3647566Z Login Succeeded 2025-12-04T09:22:18.5115077Z Command completed after 1 attempt(s). 2025-12-04T09:22:18.5166099Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:22:18.5166435Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:22:18.5166709Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:22:18.5172941Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:18.5173205Z env: 2025-12-04T09:22:18.5173372Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:18.5173557Z ##[endgroup] 2025-12-04T09:22:18.5248209Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:22:18.5248552Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:22:18.5248811Z # shellcheck disable=SC2046 2025-12-04T09:22:18.5249022Z docker stop $(docker ps -q) || true 2025-12-04T09:22:18.5249235Z # Prune all of the docker images 2025-12-04T09:22:18.5249445Z docker system prune -af 2025-12-04T09:22:18.5253929Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:18.5254167Z env: 2025-12-04T09:22:18.5254321Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:18.5254490Z ##[endgroup] 2025-12-04T09:22:18.5738447Z "docker stop" requires at least 1 argument. 2025-12-04T09:22:18.5738827Z See 'docker stop --help'. 2025-12-04T09:22:18.5738960Z 2025-12-04T09:22:18.5739087Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-12-04T09:22:18.5739322Z 2025-12-04T09:22:18.5739407Z Stop one or more running containers 2025-12-04T09:22:18.5958440Z Total reclaimed space: 0B 2025-12-04T09:22:18.6075896Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-12-04T09:22:18.6076218Z with: 2025-12-04T09:22:18.6076786Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:18.6077405Z use-custom-docker-registry: true 2025-12-04T09:22:18.6077652Z docker-build-dir: .ci/docker 2025-12-04T09:22:18.6077873Z docker-build-script: ./build.sh 2025-12-04T09:22:18.6078092Z working-directory: . 2025-12-04T09:22:18.6078338Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:18.6078614Z force-push: false 2025-12-04T09:22:18.6078784Z env: 2025-12-04T09:22:18.6078940Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:18.6079130Z ##[endgroup] 2025-12-04T09:22:18.6092724Z ##[group]Run set -ex 2025-12-04T09:22:18.6092964Z set -ex 2025-12-04T09:22:18.6093123Z  2025-12-04T09:22:18.6093415Z # If the docker build directory or the build script doesn't exist, the action will 2025-12-04T09:22:18.6093814Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-12-04T09:22:18.6094156Z # job could then download the pre-built image as usual 2025-12-04T09:22:18.6094569Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-12-04T09:22:18.6094953Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6095165Z else 2025-12-04T09:22:18.6095345Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6095626Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6095870Z  2025-12-04T09:22:18.6096211Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-12-04T09:22:18.6096581Z  exit 0 2025-12-04T09:22:18.6096725Z fi 2025-12-04T09:22:18.6096870Z  2025-12-04T09:22:18.6097094Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-12-04T09:22:18.6097456Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-12-04T09:22:18.6097888Z  # use it as it is, but first let's extract the tag 2025-12-04T09:22:18.6098188Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-12-04T09:22:18.6098500Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6098790Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6099035Z else 2025-12-04T09:22:18.6099220Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-12-04T09:22:18.6099466Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-12-04T09:22:18.6099708Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-12-04T09:22:18.6099919Z  fi 2025-12-04T09:22:18.6100207Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-12-04T09:22:18.6100559Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6100931Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6101342Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6101599Z fi 2025-12-04T09:22:18.6106159Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:18.6106390Z env: 2025-12-04T09:22:18.6106541Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:18.6106706Z REPO_NAME: pytorch 2025-12-04T09:22:18.6107290Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:18.6107805Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T09:22:18.6107989Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-12-04T09:22:18.6108223Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:18.6108471Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-12-04T09:22:18.6108659Z CUSTOM_TAG_PREFIX: 2025-12-04T09:22:18.6108820Z ##[endgroup] 2025-12-04T09:22:18.6129266Z + [[ -d .ci/docker ]] 2025-12-04T09:22:18.6129669Z + [[ -f .ci/docker/./build.sh ]] 2025-12-04T09:22:18.6129998Z + [[ true == \t\r\u\e ]] 2025-12-04T09:22:18.6130280Z + echo skip=false 2025-12-04T09:22:18.6131044Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-12-04T09:22:18.6141515Z ++ awk -F '[:,]' '{print $2}' 2025-12-04T09:22:18.6146123Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:18.6160370Z + DOCKER_TAG=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:18.6161226Z + echo docker-tag=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:18.6162123Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:18.6182472Z ##[group]Run set +e 2025-12-04T09:22:18.6182692Z set +e 2025-12-04T09:22:18.6182844Z set -x 2025-12-04T09:22:18.6182995Z  2025-12-04T09:22:18.6183140Z login() { 2025-12-04T09:22:18.6183432Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T09:22:18.6183748Z } 2025-12-04T09:22:18.6183889Z  2025-12-04T09:22:18.6184050Z retry () { 2025-12-04T09:22:18.6184230Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T09:22:18.6184424Z } 2025-12-04T09:22:18.6184558Z  2025-12-04T09:22:18.6184700Z retry login "${DOCKER_REGISTRY}" 2025-12-04T09:22:18.6184889Z  2025-12-04T09:22:18.6185136Z START_TIME=$(date +%s) 2025-12-04T09:22:18.6185335Z # Wait up to 120 minutes 2025-12-04T09:22:18.6185567Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-12-04T09:22:18.6185929Z  # Check if image already exists, if it does then skip building it 2025-12-04T09:22:18.6186236Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-12-04T09:22:18.6186454Z  exit 0 2025-12-04T09:22:18.6186609Z  fi 2025-12-04T09:22:18.6186755Z  2025-12-04T09:22:18.6187005Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-12-04T09:22:18.6187380Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-12-04T09:22:18.6187754Z  # latter, it will wait for the Docker images to become available before continuing 2025-12-04T09:22:18.6188062Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-12-04T09:22:18.6188307Z  # It's a Docker build job, let's build the image 2025-12-04T09:22:18.6188519Z  break 2025-12-04T09:22:18.6188671Z  else 2025-12-04T09:22:18.6188883Z  # It's a regular build job, wait for the image to become available 2025-12-04T09:22:18.6189121Z  sleep 300 2025-12-04T09:22:18.6189277Z  fi 2025-12-04T09:22:18.6189418Z done 2025-12-04T09:22:18.6189551Z  2025-12-04T09:22:18.6189772Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-12-04T09:22:18.6190188Z # be empty. The default action would be to continue rebuild the image 2025-12-04T09:22:18.6190490Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-12-04T09:22:18.6190763Z  # if we're on the base branch then use the parent commit 2025-12-04T09:22:18.6191011Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-12-04T09:22:18.6191210Z else 2025-12-04T09:22:18.6191412Z  # otherwise we're on a PR, so use the most recent base commit 2025-12-04T09:22:18.6191697Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-12-04T09:22:18.6191915Z fi 2025-12-04T09:22:18.6192048Z  2025-12-04T09:22:18.6192207Z if [[ -z "${MERGE_BASE}" ]]; then 2025-12-04T09:22:18.6192433Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6192638Z  2025-12-04T09:22:18.6192912Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-12-04T09:22:18.6193228Z  exit 0 2025-12-04T09:22:18.6193376Z fi 2025-12-04T09:22:18.6193505Z  2025-12-04T09:22:18.6193703Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-12-04T09:22:18.6194097Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-12-04T09:22:18.6194433Z  exit 1 2025-12-04T09:22:18.6194574Z fi 2025-12-04T09:22:18.6194714Z  2025-12-04T09:22:18.6194938Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-12-04T09:22:18.6195310Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-12-04T09:22:18.6195648Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-12-04T09:22:18.6196040Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-12-04T09:22:18.6196470Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-12-04T09:22:18.6196727Z fi 2025-12-04T09:22:18.6196867Z  2025-12-04T09:22:18.6197035Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T09:22:18.6201397Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:18.6201696Z env: 2025-12-04T09:22:18.6201850Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:18.6202025Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T09:22:18.6202259Z BASE_REVISION: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:22:18.6202830Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:18.6203533Z DOCKER_TAG: pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:18.6203970Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:18.6204206Z DOCKER_PUSH: 2025-12-04T09:22:18.6204367Z ##[endgroup] 2025-12-04T09:22:18.6229634Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:18.6231738Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:18.6232218Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:18.6236115Z + aws ecr get-login-password --region us-east-1 2025-12-04T09:22:19.0246169Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:22:19.0246948Z Configure a credential helper to remove this warning. See 2025-12-04T09:22:19.0247310Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:22:19.0247543Z 2025-12-04T09:22:19.0247612Z Login Succeeded 2025-12-04T09:22:19.0263732Z ++ date +%s 2025-12-04T09:22:19.0276221Z + START_TIME=1764840139 2025-12-04T09:22:19.0276457Z ++ date +%s 2025-12-04T09:22:19.0284963Z + [[ 1764832939 -lt 1764840139 ]] 2025-12-04T09:22:19.0285593Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:19.2615009Z { 2025-12-04T09:22:19.2615425Z "schemaVersion": 2, 2025-12-04T09:22:19.2620846Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-12-04T09:22:19.2622925Z "config": { 2025-12-04T09:22:19.2623344Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-12-04T09:22:19.2628873Z "size": 33581, 2025-12-04T09:22:19.2630946Z "digest": "sha256:40bd34edc3497e05e2150804a15313235b6426bcc4b8c19c03c3daf5524be913" 2025-12-04T09:22:19.2631383Z }, 2025-12-04T09:22:19.2631545Z "layers": [ 2025-12-04T09:22:19.2631699Z { 2025-12-04T09:22:19.2631956Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2632256Z "size": 30447951, 2025-12-04T09:22:19.2632567Z "digest": "sha256:63e5bc7682b85ae57a1221210f64d62e7a90b0a30f19af4ca734b8242ae49d63" 2025-12-04T09:22:19.2632887Z }, 2025-12-04T09:22:19.2633080Z { 2025-12-04T09:22:19.2633313Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2633596Z "size": 1554, 2025-12-04T09:22:19.2633878Z "digest": "sha256:6dc15eca51381c13be16385052dd2378ab1dce5fb77f7e7bceab34ed72e6e0e5" 2025-12-04T09:22:19.2634223Z }, 2025-12-04T09:22:19.2634377Z { 2025-12-04T09:22:19.2634608Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2634884Z "size": 313274856, 2025-12-04T09:22:19.2635162Z "digest": "sha256:459f4df18f07c473cb662e8033f9dab9f68c9ac71608767ad055999fae6b31e2" 2025-12-04T09:22:19.2635457Z }, 2025-12-04T09:22:19.2635593Z { 2025-12-04T09:22:19.2635808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2636097Z "size": 787, 2025-12-04T09:22:19.2636380Z "digest": "sha256:82108541691974f1cc469d0ec89a048191b8de4dc36bf9a0184a4bd50120b69d" 2025-12-04T09:22:19.2636709Z }, 2025-12-04T09:22:19.2636835Z { 2025-12-04T09:22:19.2637056Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2637331Z "size": 106, 2025-12-04T09:22:19.2637643Z "digest": "sha256:3738646b3d92a3f1b6529ba588c00216c306c997ad77730f207c7107c2e5aff3" 2025-12-04T09:22:19.2639339Z }, 2025-12-04T09:22:19.2639477Z { 2025-12-04T09:22:19.2639690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2639953Z "size": 704, 2025-12-04T09:22:19.2640215Z "digest": "sha256:aa9a10a37b3945f69568695fa4a9321333654a69dd44c55eff4e26bf9bffcad1" 2025-12-04T09:22:19.2640500Z }, 2025-12-04T09:22:19.2640631Z { 2025-12-04T09:22:19.2640840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2641100Z "size": 1218, 2025-12-04T09:22:19.2641366Z "digest": "sha256:55bac08f3e18897f61d3e7f28ada6547b05b3c36f559a882656834d43de61478" 2025-12-04T09:22:19.2641661Z }, 2025-12-04T09:22:19.2641795Z { 2025-12-04T09:22:19.2642006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2642271Z "size": 485, 2025-12-04T09:22:19.2642545Z "digest": "sha256:acf6468f6aadc3613e3b8fe73c016ce8d6f47f01e0cfee3a1015d0335c24d691" 2025-12-04T09:22:19.2642834Z }, 2025-12-04T09:22:19.2642967Z { 2025-12-04T09:22:19.2643185Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2643452Z "size": 110362453, 2025-12-04T09:22:19.2643735Z "digest": "sha256:c0c31a0e69b798875a8d2c05d01a6861b2fe623e1970bf7faf822e7df6691d2c" 2025-12-04T09:22:19.2644028Z }, 2025-12-04T09:22:19.2644156Z { 2025-12-04T09:22:19.2644374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2644645Z "size": 4961, 2025-12-04T09:22:19.2644907Z "digest": "sha256:0f7d0dc70d2d944cc1c8a592f051d3f28d2b062c79166713fb7cc9225b4c97c0" 2025-12-04T09:22:19.2645200Z }, 2025-12-04T09:22:19.2645431Z { 2025-12-04T09:22:19.2645646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2645957Z "size": 1756, 2025-12-04T09:22:19.2646229Z "digest": "sha256:58c4d15d3bc18a789eb0d1a13f68df0b1d6cea66697fe63adf11b53fec74ba18" 2025-12-04T09:22:19.2646533Z }, 2025-12-04T09:22:19.2646672Z { 2025-12-04T09:22:19.2646877Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2647144Z "size": 724, 2025-12-04T09:22:19.2647419Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:22:19.2647717Z }, 2025-12-04T09:22:19.2647844Z { 2025-12-04T09:22:19.2648060Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2648331Z "size": 543, 2025-12-04T09:22:19.2648591Z "digest": "sha256:d2472a74103d877e6532d6409a2ac9d54db6672b5d3002be011fdbbd8d360c49" 2025-12-04T09:22:19.2648883Z }, 2025-12-04T09:22:19.2649017Z { 2025-12-04T09:22:19.2649227Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2649495Z "size": 3353192867, 2025-12-04T09:22:19.2649780Z "digest": "sha256:d1fba72936888ea8f8e4c7ad015bfcf3285017b95f3d72795d5b6a0b0ded36fc" 2025-12-04T09:22:19.2650227Z }, 2025-12-04T09:22:19.2650370Z { 2025-12-04T09:22:19.2650587Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2650858Z "size": 32, 2025-12-04T09:22:19.2651125Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2651425Z }, 2025-12-04T09:22:19.2651711Z { 2025-12-04T09:22:19.2651930Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2669459Z "size": 397, 2025-12-04T09:22:19.2669794Z "digest": "sha256:6a9b03ce41a77be16366fba2c6ca0aabefe42ee4879ab5d088b312970e830770" 2025-12-04T09:22:19.2670128Z }, 2025-12-04T09:22:19.2670277Z { 2025-12-04T09:22:19.2670531Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2670820Z "size": 237420, 2025-12-04T09:22:19.2671108Z "digest": "sha256:3519ac15be7978a14201f5d431b6472ed30382cde8628c6401c9db98ccd1a871" 2025-12-04T09:22:19.2671412Z }, 2025-12-04T09:22:19.2671547Z { 2025-12-04T09:22:19.2671775Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2672190Z "size": 231, 2025-12-04T09:22:19.2672884Z "digest": "sha256:dabb51b819a06823e5a1c8d4f9e29b689f38b19ee1e90da40075bd1593b8d0fb" 2025-12-04T09:22:19.2673194Z }, 2025-12-04T09:22:19.2673336Z { 2025-12-04T09:22:19.2673552Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2673824Z "size": 2967944, 2025-12-04T09:22:19.2674112Z "digest": "sha256:8f36833a24d02d47bf18bdc573adbb45afb8f5f06886da9bd671a1a33e3007bd" 2025-12-04T09:22:19.2674409Z }, 2025-12-04T09:22:19.2674548Z { 2025-12-04T09:22:19.2674765Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2675029Z "size": 1472, 2025-12-04T09:22:19.2675311Z "digest": "sha256:ab53c5a853a52de13f11fbb026aaf469754adbe52b02e96f4fbdc05916585244" 2025-12-04T09:22:19.2675608Z }, 2025-12-04T09:22:19.2675743Z { 2025-12-04T09:22:19.2675950Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2676211Z "size": 482, 2025-12-04T09:22:19.2676460Z + exit 0 2025-12-04T09:22:19.2676718Z "digest": "sha256:025a0e5e6ac19cb2d27c9423a5d019a34ea7d4004d426d6b890693ccfe9f54d3" 2025-12-04T09:22:19.2677015Z }, 2025-12-04T09:22:19.2677151Z { 2025-12-04T09:22:19.2677360Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2677624Z "size": 200, 2025-12-04T09:22:19.2677901Z "digest": "sha256:ce3394c8f2109f4538f6e7f2c39f685d0ba028aacac33ddacbb8ef9fd349702e" 2025-12-04T09:22:19.2678197Z }, 2025-12-04T09:22:19.2678341Z { 2025-12-04T09:22:19.2678542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2678914Z "size": 608, 2025-12-04T09:22:19.2679162Z "digest": "sha256:a5c3888c3a0c5812efbfe307d18de826363900fa8d8097c1ee84a72630aa067b" 2025-12-04T09:22:19.2679436Z }, 2025-12-04T09:22:19.2679568Z { 2025-12-04T09:22:19.2679760Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2680005Z "size": 225, 2025-12-04T09:22:19.2680256Z "digest": "sha256:ed902a3a4e3b916aa869b612baf9cd36745b671b6781d19e3b4c3fadc2f513f5" 2025-12-04T09:22:19.2680527Z }, 2025-12-04T09:22:19.2680658Z { 2025-12-04T09:22:19.2680860Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2681092Z "size": 829, 2025-12-04T09:22:19.2681345Z "digest": "sha256:b4e1efca22beb475459a14f4a571a2512c2b1ed4b24e66bd3f82f1681dc5b1bc" 2025-12-04T09:22:19.2681615Z }, 2025-12-04T09:22:19.2681734Z { 2025-12-04T09:22:19.2681930Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2682166Z "size": 32, 2025-12-04T09:22:19.2682421Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2682686Z }, 2025-12-04T09:22:19.2682811Z { 2025-12-04T09:22:19.2683006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2683240Z "size": 104, 2025-12-04T09:22:19.2683489Z "digest": "sha256:79fcfd297d9fcfe2810595fc2c6ba503293b59046bc192bb0620a64d9bdff778" 2025-12-04T09:22:19.2683765Z }, 2025-12-04T09:22:19.2683882Z { 2025-12-04T09:22:19.2684078Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2684319Z "size": 1495, 2025-12-04T09:22:19.2684562Z "digest": "sha256:7bd89c134b49f83cfb0725df3615556b60af380b932186c5abd001232b4597cf" 2025-12-04T09:22:19.2684830Z }, 2025-12-04T09:22:19.2684951Z { 2025-12-04T09:22:19.2685139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2685378Z "size": 458790783, 2025-12-04T09:22:19.2685645Z "digest": "sha256:b7d40b4fd1b9375cfd45d5c593ec0cccc21c80d667ea79f4effb09f04b0705b0" 2025-12-04T09:22:19.2685921Z }, 2025-12-04T09:22:19.2686040Z { 2025-12-04T09:22:19.2686241Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2686482Z "size": 163, 2025-12-04T09:22:19.2686732Z "digest": "sha256:dd1cdd87320d29d4ca09686ab00b76a396efcc6f3ea6d0bfd1f7922e46336ca6" 2025-12-04T09:22:19.2687095Z }, 2025-12-04T09:22:19.2687222Z { 2025-12-04T09:22:19.2687423Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2687670Z "size": 347, 2025-12-04T09:22:19.2687933Z "digest": "sha256:c21a6e1cd03a718ebfb927ab114ebbbee0b57a93908d0265d24b02cf34c0fee3" 2025-12-04T09:22:19.2688211Z }, 2025-12-04T09:22:19.2688343Z { 2025-12-04T09:22:19.2688549Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2688798Z "size": 32, 2025-12-04T09:22:19.2689052Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2689353Z }, 2025-12-04T09:22:19.2689486Z { 2025-12-04T09:22:19.2689692Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2689953Z "size": 106, 2025-12-04T09:22:19.2690227Z "digest": "sha256:0aa3ea0b275426eac8eff6b26ba17c168ae70d1baa5a7845e61d41bd01a9ff43" 2025-12-04T09:22:19.2690517Z }, 2025-12-04T09:22:19.2690655Z { 2025-12-04T09:22:19.2690868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2691118Z "size": 426, 2025-12-04T09:22:19.2691396Z "digest": "sha256:74e0bdbb05d31425d7a2f40ff0cb2423abaf90f1ed56c4b69f43e8d92c569379" 2025-12-04T09:22:19.2691814Z }, 2025-12-04T09:22:19.2691947Z { 2025-12-04T09:22:19.2692167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2692440Z "size": 19309400, 2025-12-04T09:22:19.2692736Z "digest": "sha256:ba3aee7dcf03efbd1e7b1611c8f65f385c94b2572d15c099a1535f27b72509f7" 2025-12-04T09:22:19.2693094Z }, 2025-12-04T09:22:19.2693244Z { 2025-12-04T09:22:19.2693463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2693718Z "size": 108, 2025-12-04T09:22:19.2693985Z "digest": "sha256:8c8d59b8759d40399b07b019a46fc05b9ccabe2b9276f49eea203fd6908ff334" 2025-12-04T09:22:19.2694280Z }, 2025-12-04T09:22:19.2694407Z { 2025-12-04T09:22:19.2694630Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2694891Z "size": 827, 2025-12-04T09:22:19.2695145Z "digest": "sha256:3c960865867b66327e316bf682adabd1867dd9a41cc0aee1507ee8e35fa614a9" 2025-12-04T09:22:19.2695439Z }, 2025-12-04T09:22:19.2695571Z { 2025-12-04T09:22:19.2695786Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2696040Z "size": 724, 2025-12-04T09:22:19.2696315Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:22:19.2696615Z }, 2025-12-04T09:22:19.2696740Z { 2025-12-04T09:22:19.2696957Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2697217Z "size": 149, 2025-12-04T09:22:19.2697478Z "digest": "sha256:074db79e3832af094ddad6e5c5649347b62e8ba0365229feced3ed0263a0c611" 2025-12-04T09:22:19.2697774Z }, 2025-12-04T09:22:19.2697909Z { 2025-12-04T09:22:19.2698115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2698382Z "size": 137, 2025-12-04T09:22:19.2698649Z "digest": "sha256:226748f4ff23ba2aef33619c3a639fb44865d5f4515c4b45d4196f4e0e77b187" 2025-12-04T09:22:19.2698934Z }, 2025-12-04T09:22:19.2699067Z { 2025-12-04T09:22:19.2699287Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2699551Z "size": 140, 2025-12-04T09:22:19.2699811Z "digest": "sha256:75d817336f1df7579f5b4be29625c53d594cb78c776f5454cc69438bb1bb0dc9" 2025-12-04T09:22:19.2700104Z }, 2025-12-04T09:22:19.2700242Z { 2025-12-04T09:22:19.2700454Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2700722Z "size": 18890247472, 2025-12-04T09:22:19.2701004Z "digest": "sha256:850870a17b94a4d7c5e55b287cc0b1ddd0982845e11f055765dc358277cdfc54" 2025-12-04T09:22:19.2701284Z }, 2025-12-04T09:22:19.2701417Z { 2025-12-04T09:22:19.2701629Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2701933Z "size": 222, 2025-12-04T09:22:19.2702195Z "digest": "sha256:e6225129924f0a66acaf35efd792901351da58116629962669d9c3acaadbab6e" 2025-12-04T09:22:19.2702480Z }, 2025-12-04T09:22:19.2702607Z { 2025-12-04T09:22:19.2702819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2703082Z "size": 255, 2025-12-04T09:22:19.2703352Z "digest": "sha256:5cd6b9ec5c3c4a1cf2e2e7a1794aaae173eb5a291b9899bb733e5f36d866fab0" 2025-12-04T09:22:19.2703644Z }, 2025-12-04T09:22:19.2703778Z { 2025-12-04T09:22:19.2703994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2704246Z "size": 32, 2025-12-04T09:22:19.2704517Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2704813Z }, 2025-12-04T09:22:19.2704940Z { 2025-12-04T09:22:19.2705151Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2705409Z "size": 106, 2025-12-04T09:22:19.2705678Z "digest": "sha256:613d2b1f3db875d1d695081b44aefbf94eeb7815d794b28dd17ec623d3f9dad5" 2025-12-04T09:22:19.2705972Z }, 2025-12-04T09:22:19.2706105Z { 2025-12-04T09:22:19.2706314Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2706574Z "size": 312293443, 2025-12-04T09:22:19.2706851Z "digest": "sha256:388169fffe8a06396f4e8d6fc51d07a83e08f04836995ae28c2e13960cb89fab" 2025-12-04T09:22:19.2707139Z }, 2025-12-04T09:22:19.2707265Z { 2025-12-04T09:22:19.2707479Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2707819Z "size": 6628443345, 2025-12-04T09:22:19.2708099Z "digest": "sha256:6ca498e78b7b821a815bc7c14f057d52f7d9ce8dda8bca0945e6f522c7d208e1" 2025-12-04T09:22:19.2708391Z }, 2025-12-04T09:22:19.2708525Z { 2025-12-04T09:22:19.2708730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2708991Z "size": 129, 2025-12-04T09:22:19.2709256Z "digest": "sha256:952b753ec7be39e1967f95afb87a880712b6425610adcda2afe3b2211cc8de12" 2025-12-04T09:22:19.2709540Z }, 2025-12-04T09:22:19.2709675Z { 2025-12-04T09:22:19.2709887Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2710157Z "size": 880, 2025-12-04T09:22:19.2710435Z "digest": "sha256:5b6f6de4bdd1c96aa8704c326fb27fa7465ebf388f0748ee56d9f521bb6b6697" 2025-12-04T09:22:19.2710738Z }, 2025-12-04T09:22:19.2710871Z { 2025-12-04T09:22:19.2711074Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2711329Z "size": 724, 2025-12-04T09:22:19.2711611Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:22:19.2711900Z }, 2025-12-04T09:22:19.2712029Z { 2025-12-04T09:22:19.2712230Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2712466Z "size": 139, 2025-12-04T09:22:19.2712724Z "digest": "sha256:fa23d9952f3cea91af7b5ffce93e64d90d98aefb838aa8a1fe4a6b40cd0eab91" 2025-12-04T09:22:19.2713010Z }, 2025-12-04T09:22:19.2713130Z { 2025-12-04T09:22:19.2713328Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2713573Z "size": 32, 2025-12-04T09:22:19.2713823Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2714096Z }, 2025-12-04T09:22:19.2714230Z { 2025-12-04T09:22:19.2714430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2714669Z "size": 161, 2025-12-04T09:22:19.2714933Z "digest": "sha256:c2aa9ea4d09ac3edb41e48cdc892fafe72a5cddc98ebdf67d978c6f8d63cd7d2" 2025-12-04T09:22:19.2715222Z }, 2025-12-04T09:22:19.2715341Z { 2025-12-04T09:22:19.2715542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2715791Z "size": 1011, 2025-12-04T09:22:19.2716057Z "digest": "sha256:a9bbc9e426d367e15c7f0c1faae6417526e219fa8188ae17b9e6c2c3b8083bdc" 2025-12-04T09:22:19.2716372Z }, 2025-12-04T09:22:19.2716503Z { 2025-12-04T09:22:19.2716703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2716941Z "size": 724, 2025-12-04T09:22:19.2717201Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:22:19.2717483Z }, 2025-12-04T09:22:19.2717603Z { 2025-12-04T09:22:19.2717806Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2718051Z "size": 135, 2025-12-04T09:22:19.2718297Z "digest": "sha256:82f2371f6dc2c2797baa60e672fd78bb04909cd0248c52cae058a7fd8f215a4f" 2025-12-04T09:22:19.2718579Z }, 2025-12-04T09:22:19.2718704Z { 2025-12-04T09:22:19.2718896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2719138Z "size": 32, 2025-12-04T09:22:19.2719389Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2719663Z }, 2025-12-04T09:22:19.2719784Z { 2025-12-04T09:22:19.2719986Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2720229Z "size": 158, 2025-12-04T09:22:19.2720471Z "digest": "sha256:c87301921afac5ba5745add04b44f57bf31c551bf48a2574d4424dab64c25ce2" 2025-12-04T09:22:19.2720746Z }, 2025-12-04T09:22:19.2720873Z { 2025-12-04T09:22:19.2721067Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2721311Z "size": 603, 2025-12-04T09:22:19.2721571Z "digest": "sha256:384d566a822f4defa0ecf9c7d37b6d7a5fc5eaab415fe1d3cc02b190ddc19e71" 2025-12-04T09:22:19.2721846Z }, 2025-12-04T09:22:19.2722015Z { 2025-12-04T09:22:19.2722219Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2722466Z "size": 724, 2025-12-04T09:22:19.2722719Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T09:22:19.2723000Z }, 2025-12-04T09:22:19.2723129Z { 2025-12-04T09:22:19.2723322Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2723573Z "size": 155, 2025-12-04T09:22:19.2723828Z "digest": "sha256:32ddcdd5dd6920acb12ca95671a164346308ec7743afbe8a333142733c42bafd" 2025-12-04T09:22:19.2724100Z }, 2025-12-04T09:22:19.2724226Z { 2025-12-04T09:22:19.2724427Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2724665Z "size": 32, 2025-12-04T09:22:19.2724920Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2725199Z }, 2025-12-04T09:22:19.2725318Z { 2025-12-04T09:22:19.2725521Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2725765Z "size": 188, 2025-12-04T09:22:19.2726022Z "digest": "sha256:a116a32ceaa15bbd55b0f4281fb0040de221c62e29659b23dd71ecd2382a969a" 2025-12-04T09:22:19.2726295Z }, 2025-12-04T09:22:19.2726422Z { 2025-12-04T09:22:19.2726621Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2726864Z "size": 1371, 2025-12-04T09:22:19.2727126Z "digest": "sha256:f6fbbbe6067e383cadb530c3f53b113e9826843fb2348ea1f9a000a4790fa822" 2025-12-04T09:22:19.2727403Z }, 2025-12-04T09:22:19.2727523Z { 2025-12-04T09:22:19.2727722Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2727968Z "size": 32, 2025-12-04T09:22:19.2728214Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2728493Z }, 2025-12-04T09:22:19.2728621Z { 2025-12-04T09:22:19.2728819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2729076Z "size": 137, 2025-12-04T09:22:19.2729338Z "digest": "sha256:648018658875ac3356b277abe3d98df9a8b7a0345f131598c07f5e9f9318b740" 2025-12-04T09:22:19.2729632Z }, 2025-12-04T09:22:19.2729756Z { 2025-12-04T09:22:19.2729965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2730232Z "size": 528, 2025-12-04T09:22:19.2730560Z "digest": "sha256:5e2f7eac20ad5128eebe0ba7dbd08111d28e7f65c26c7fea7cd6dc7a2c0725b9" 2025-12-04T09:22:19.2730861Z }, 2025-12-04T09:22:19.2730994Z { 2025-12-04T09:22:19.2731200Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2731458Z "size": 32, 2025-12-04T09:22:19.2731831Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2732135Z }, 2025-12-04T09:22:19.2732276Z { 2025-12-04T09:22:19.2732495Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2732768Z "size": 104, 2025-12-04T09:22:19.2733039Z "digest": "sha256:0d1243af7593a042ff443492a4cac70f817dc8c56ac45a0c138d102f40c1cc07" 2025-12-04T09:22:19.2733335Z }, 2025-12-04T09:22:19.2733472Z { 2025-12-04T09:22:19.2733680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2733946Z "size": 435, 2025-12-04T09:22:19.2734220Z "digest": "sha256:1f6d5b941ea1f6f07c190f5ca1d4a5172e4d696ad597ae58dcd37b297f60b556" 2025-12-04T09:22:19.2734520Z }, 2025-12-04T09:22:19.2734653Z { 2025-12-04T09:22:19.2734869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2735125Z "size": 32, 2025-12-04T09:22:19.2735390Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2735694Z }, 2025-12-04T09:22:19.2735821Z { 2025-12-04T09:22:19.2736031Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2736295Z "size": 107, 2025-12-04T09:22:19.2736624Z "digest": "sha256:4444f70d73cefec9f161242e9def76078af5abc7a3ad3e5d6e7d4f1efed07939" 2025-12-04T09:22:19.2736917Z }, 2025-12-04T09:22:19.2737053Z { 2025-12-04T09:22:19.2737265Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2737513Z "size": 1896, 2025-12-04T09:22:19.2737794Z "digest": "sha256:bf45550ddbad2524852930466f963e432429f2b86f7f6ccf0415fc56980004ce" 2025-12-04T09:22:19.2738084Z }, 2025-12-04T09:22:19.2738211Z { 2025-12-04T09:22:19.2738423Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2738685Z "size": 245588924, 2025-12-04T09:22:19.2738959Z "digest": "sha256:30fff4f4bad325a3a2d91463ca6702c411392e5b0646a5cd4f47fcae56c55639" 2025-12-04T09:22:19.2739252Z }, 2025-12-04T09:22:19.2739389Z { 2025-12-04T09:22:19.2739592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2739846Z "size": 106, 2025-12-04T09:22:19.2740117Z "digest": "sha256:cca4db1ba155a3afd1fd645cd3223e9a4de9c785f8559387f21da187672d9e9e" 2025-12-04T09:22:19.2740415Z }, 2025-12-04T09:22:19.2740539Z { 2025-12-04T09:22:19.2740755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2740995Z "size": 165, 2025-12-04T09:22:19.2741241Z "digest": "sha256:f0ffa379f4eba2f3ddc323f45f226f39aed5a7e93880884a97bd4d6a91864dc1" 2025-12-04T09:22:19.2741521Z }, 2025-12-04T09:22:19.2741644Z { 2025-12-04T09:22:19.2741836Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2742080Z "size": 7942, 2025-12-04T09:22:19.2742327Z "digest": "sha256:df645e678f95db120f96e4c4c66400d315ed9214d9085216b417348cf682d43a" 2025-12-04T09:22:19.2742593Z }, 2025-12-04T09:22:19.2742716Z { 2025-12-04T09:22:19.2742917Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2743152Z "size": 8076, 2025-12-04T09:22:19.2743407Z "digest": "sha256:52ad6c17d10308710fb2dc855f911246b7a369ccf5c3142cb3ad7fb182f708b8" 2025-12-04T09:22:19.2743680Z }, 2025-12-04T09:22:19.2743803Z { 2025-12-04T09:22:19.2743997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2744238Z "size": 302, 2025-12-04T09:22:19.2744490Z "digest": "sha256:34408374e32c70dbcfd7e0516edaa8698c25e6e3dbcd920978f2b3a801bbe659" 2025-12-04T09:22:19.2744761Z }, 2025-12-04T09:22:19.2744931Z { 2025-12-04T09:22:19.2745136Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2745375Z "size": 32, 2025-12-04T09:22:19.2745633Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2745913Z }, 2025-12-04T09:22:19.2746034Z { 2025-12-04T09:22:19.2746237Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2746480Z "size": 108, 2025-12-04T09:22:19.2746736Z "digest": "sha256:27803b661d9e2cb1ce49c0a100ce824fbcbdd1f3a3ceda4ed5affbcb07ac3be4" 2025-12-04T09:22:19.2747015Z }, 2025-12-04T09:22:19.2747142Z { 2025-12-04T09:22:19.2747344Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2747589Z "size": 54145699, 2025-12-04T09:22:19.2747861Z "digest": "sha256:07dafc893dea952d7677124b42e90b29b167fabe81e8a9d8b3fb4b0aa11b66ec" 2025-12-04T09:22:19.2748143Z }, 2025-12-04T09:22:19.2748263Z { 2025-12-04T09:22:19.2748463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T09:22:19.2748707Z "size": 32, 2025-12-04T09:22:19.2748950Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T09:22:19.2749229Z } 2025-12-04T09:22:19.2749354Z ] 2025-12-04T09:22:19.2749479Z } 2025-12-04T09:22:19.2774092Z ##[group]Run set -eux 2025-12-04T09:22:19.2774326Z set -eux 2025-12-04T09:22:19.2774603Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-12-04T09:22:19.2775472Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-12-04T09:22:19.2780875Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:19.2781148Z env: 2025-12-04T09:22:19.2781321Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:19.2781490Z ##[endgroup] 2025-12-04T09:22:19.2805394Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-12-04T09:22:19.2806133Z + jq -r .docker_hub_readonly_token 2025-12-04T09:22:19.2807333Z + docker login --username pytorchbot --password-stdin 2025-12-04T09:22:19.2807903Z + jq --raw-output .SecretString 2025-12-04T09:22:19.7271426Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:22:19.7272176Z Configure a credential helper to remove this warning. See 2025-12-04T09:22:19.7272698Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:22:19.7272956Z 2025-12-04T09:22:19.7273023Z Login Succeeded 2025-12-04T09:22:19.7344453Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T09:22:19.7344697Z tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T09:22:19.7344952Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-12-04T09:22:19.7349619Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:19.7349865Z env: 2025-12-04T09:22:19.7350019Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:19.7350535Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:19.7351047Z ##[endgroup] 2025-12-04T09:22:19.7379728Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:19.7410111Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-12-04T09:22:19.7410361Z with: 2025-12-04T09:22:19.7410841Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:19.7411684Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:19.7412016Z env: 2025-12-04T09:22:19.7412204Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:19.7412558Z ##[endgroup] 2025-12-04T09:22:19.7422799Z ##[group]Run set -x 2025-12-04T09:22:19.7422974Z set -x 2025-12-04T09:22:19.7423114Z set +e 2025-12-04T09:22:19.7423256Z  2025-12-04T09:22:19.7423395Z login() { 2025-12-04T09:22:19.7423674Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T09:22:19.7423964Z } 2025-12-04T09:22:19.7424098Z  2025-12-04T09:22:19.7424258Z retry () { 2025-12-04T09:22:19.7424425Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T09:22:19.7424612Z } 2025-12-04T09:22:19.7424742Z  2025-12-04T09:22:19.7424886Z retry login "${DOCKER_REGISTRY}" 2025-12-04T09:22:19.7425060Z  2025-12-04T09:22:19.7425337Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-12-04T09:22:19.7425702Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-12-04T09:22:19.7425917Z  2025-12-04T09:22:19.7426049Z set -e 2025-12-04T09:22:19.7426256Z # ignore output since only exit code is used for conditional 2025-12-04T09:22:19.7426535Z # only pull docker image if it's not available locally 2025-12-04T09:22:19.7426831Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-12-04T09:22:19.7427117Z  retry docker pull "${DOCKER_IMAGE}" 2025-12-04T09:22:19.7427308Z fi 2025-12-04T09:22:19.7430999Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:22:19.7431227Z env: 2025-12-04T09:22:19.7431378Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:22:19.7431866Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:19.7432422Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:19.7432664Z ##[endgroup] 2025-12-04T09:22:19.7455438Z + set +e 2025-12-04T09:22:19.7455914Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:19.7456369Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:19.7456804Z + aws ecr get-login-password --region us-east-1 2025-12-04T09:22:19.7457276Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T09:22:20.1703285Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T09:22:20.1703749Z Configure a credential helper to remove this warning. See 2025-12-04T09:22:20.1704554Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T09:22:20.1705442Z 2025-12-04T09:22:20.1706110Z Login Succeeded 2025-12-04T09:22:20.1726276Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:20.1727362Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-12-04T09:22:20.3572510Z + IMAGE_SIZE=29010.187264442444 2025-12-04T09:22:20.3572844Z Compressed size of image in MB: 29010.187264442444 2025-12-04T09:22:20.3573153Z + echo 'Compressed size of image in MB: 29010.187264442444' 2025-12-04T09:22:20.3573418Z + set -e 2025-12-04T09:22:20.3575015Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:20.3768664Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:20.3769563Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:22:20.7382740Z pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a: Pulling from pytorch/ci-image 2025-12-04T09:22:20.7384292Z 63e5bc7682b8: Pulling fs layer 2025-12-04T09:22:20.7384582Z 6dc15eca5138: Pulling fs layer 2025-12-04T09:22:20.7384840Z 459f4df18f07: Pulling fs layer 2025-12-04T09:22:20.7385128Z 821085416919: Pulling fs layer 2025-12-04T09:22:20.7385357Z 3738646b3d92: Pulling fs layer 2025-12-04T09:22:20.7385559Z aa9a10a37b39: Pulling fs layer 2025-12-04T09:22:20.7385846Z 55bac08f3e18: Pulling fs layer 2025-12-04T09:22:20.7391189Z acf6468f6aad: Pulling fs layer 2025-12-04T09:22:20.7396414Z c0c31a0e69b7: Pulling fs layer 2025-12-04T09:22:20.7400894Z 0f7d0dc70d2d: Pulling fs layer 2025-12-04T09:22:20.7403174Z 58c4d15d3bc1: Pulling fs layer 2025-12-04T09:22:20.7403484Z 2f1cc47b61e1: Pulling fs layer 2025-12-04T09:22:20.7403919Z d2472a74103d: Pulling fs layer 2025-12-04T09:22:20.7404130Z d1fba7293688: Pulling fs layer 2025-12-04T09:22:20.7404343Z 4f4fb700ef54: Pulling fs layer 2025-12-04T09:22:20.7404554Z 6a9b03ce41a7: Pulling fs layer 2025-12-04T09:22:20.7404759Z 3519ac15be79: Pulling fs layer 2025-12-04T09:22:20.7404975Z dabb51b819a0: Pulling fs layer 2025-12-04T09:22:20.7405179Z 8f36833a24d0: Pulling fs layer 2025-12-04T09:22:20.7405350Z ab53c5a853a5: Pulling fs layer 2025-12-04T09:22:20.7405529Z 025a0e5e6ac1: Pulling fs layer 2025-12-04T09:22:20.7405704Z ce3394c8f210: Pulling fs layer 2025-12-04T09:22:20.7405871Z a5c3888c3a0c: Pulling fs layer 2025-12-04T09:22:20.7406051Z ed902a3a4e3b: Pulling fs layer 2025-12-04T09:22:20.7406228Z b4e1efca22be: Pulling fs layer 2025-12-04T09:22:20.7406395Z 79fcfd297d9f: Pulling fs layer 2025-12-04T09:22:20.7406568Z 7bd89c134b49: Pulling fs layer 2025-12-04T09:22:20.7406743Z b7d40b4fd1b9: Pulling fs layer 2025-12-04T09:22:20.7406911Z dd1cdd87320d: Pulling fs layer 2025-12-04T09:22:20.7407087Z c21a6e1cd03a: Pulling fs layer 2025-12-04T09:22:20.7407261Z 0aa3ea0b2754: Pulling fs layer 2025-12-04T09:22:20.7407427Z 74e0bdbb05d3: Pulling fs layer 2025-12-04T09:22:20.7407603Z ba3aee7dcf03: Pulling fs layer 2025-12-04T09:22:20.7407787Z 8c8d59b8759d: Pulling fs layer 2025-12-04T09:22:20.7407961Z 3c960865867b: Pulling fs layer 2025-12-04T09:22:20.7408125Z 074db79e3832: Pulling fs layer 2025-12-04T09:22:20.7458503Z 226748f4ff23: Pulling fs layer 2025-12-04T09:22:20.7458771Z 75d817336f1d: Pulling fs layer 2025-12-04T09:22:20.7459291Z 850870a17b94: Pulling fs layer 2025-12-04T09:22:20.7459550Z e6225129924f: Pulling fs layer 2025-12-04T09:22:20.7459777Z 5cd6b9ec5c3c: Pulling fs layer 2025-12-04T09:22:20.7459978Z 613d2b1f3db8: Pulling fs layer 2025-12-04T09:22:20.7460163Z 388169fffe8a: Pulling fs layer 2025-12-04T09:22:20.7460352Z 6ca498e78b7b: Pulling fs layer 2025-12-04T09:22:20.7460543Z 952b753ec7be: Pulling fs layer 2025-12-04T09:22:20.7460730Z 5b6f6de4bdd1: Pulling fs layer 2025-12-04T09:22:20.7460924Z fa23d9952f3c: Pulling fs layer 2025-12-04T09:22:20.7461111Z c2aa9ea4d09a: Pulling fs layer 2025-12-04T09:22:20.7461490Z a9bbc9e426d3: Pulling fs layer 2025-12-04T09:22:20.7461682Z 82f2371f6dc2: Pulling fs layer 2025-12-04T09:22:20.7461878Z c87301921afa: Pulling fs layer 2025-12-04T09:22:20.7462069Z 384d566a822f: Pulling fs layer 2025-12-04T09:22:20.7462252Z 32ddcdd5dd69: Pulling fs layer 2025-12-04T09:22:20.7462440Z a116a32ceaa1: Pulling fs layer 2025-12-04T09:22:20.7462632Z f6fbbbe6067e: Pulling fs layer 2025-12-04T09:22:20.7462800Z 648018658875: Pulling fs layer 2025-12-04T09:22:20.7463024Z 5e2f7eac20ad: Pulling fs layer 2025-12-04T09:22:20.7463451Z 0d1243af7593: Pulling fs layer 2025-12-04T09:22:20.7463641Z 1f6d5b941ea1: Pulling fs layer 2025-12-04T09:22:20.7463824Z 4444f70d73ce: Pulling fs layer 2025-12-04T09:22:20.7464013Z bf45550ddbad: Pulling fs layer 2025-12-04T09:22:20.7464207Z 30fff4f4bad3: Pulling fs layer 2025-12-04T09:22:20.7464393Z cca4db1ba155: Pulling fs layer 2025-12-04T09:22:20.7464587Z f0ffa379f4eb: Pulling fs layer 2025-12-04T09:22:20.7464780Z df645e678f95: Pulling fs layer 2025-12-04T09:22:20.7464960Z 52ad6c17d103: Pulling fs layer 2025-12-04T09:22:20.7465153Z 34408374e32c: Pulling fs layer 2025-12-04T09:22:20.7465398Z 27803b661d9e: Pulling fs layer 2025-12-04T09:22:20.7465580Z 07dafc893dea: Pulling fs layer 2025-12-04T09:22:20.7465772Z 7bd89c134b49: Waiting 2025-12-04T09:22:20.7465947Z b7d40b4fd1b9: Waiting 2025-12-04T09:22:20.7466106Z dd1cdd87320d: Waiting 2025-12-04T09:22:20.7466270Z 82f2371f6dc2: Waiting 2025-12-04T09:22:20.7466430Z 850870a17b94: Waiting 2025-12-04T09:22:20.7466591Z c87301921afa: Waiting 2025-12-04T09:22:20.7466750Z 384d566a822f: Waiting 2025-12-04T09:22:20.7466910Z e6225129924f: Waiting 2025-12-04T09:22:20.7467073Z 5cd6b9ec5c3c: Waiting 2025-12-04T09:22:20.7467227Z f6fbbbe6067e: Waiting 2025-12-04T09:22:20.7467389Z 52ad6c17d103: Waiting 2025-12-04T09:22:20.7467549Z 34408374e32c: Waiting 2025-12-04T09:22:20.7467700Z 648018658875: Waiting 2025-12-04T09:22:20.7467858Z 952b753ec7be: Waiting 2025-12-04T09:22:20.7468018Z dabb51b819a0: Waiting 2025-12-04T09:22:20.7468170Z 58c4d15d3bc1: Waiting 2025-12-04T09:22:20.7468330Z 32ddcdd5dd69: Waiting 2025-12-04T09:22:20.7468493Z 8f36833a24d0: Waiting 2025-12-04T09:22:20.7468645Z 074db79e3832: Waiting 2025-12-04T09:22:20.7468805Z a116a32ceaa1: Waiting 2025-12-04T09:22:20.7468964Z ab53c5a853a5: Waiting 2025-12-04T09:22:20.7469117Z 55bac08f3e18: Waiting 2025-12-04T09:22:20.7469278Z 5e2f7eac20ad: Waiting 2025-12-04T09:22:20.7469440Z 3519ac15be79: Waiting 2025-12-04T09:22:20.7469594Z 2f1cc47b61e1: Waiting 2025-12-04T09:22:20.7469753Z d2472a74103d: Waiting 2025-12-04T09:22:20.7469922Z a9bbc9e426d3: Waiting 2025-12-04T09:22:20.7470128Z a5c3888c3a0c: Waiting 2025-12-04T09:22:20.7470296Z 30fff4f4bad3: Waiting 2025-12-04T09:22:20.7470459Z ed902a3a4e3b: Waiting 2025-12-04T09:22:20.7470623Z ce3394c8f210: Waiting 2025-12-04T09:22:20.7470776Z 79fcfd297d9f: Waiting 2025-12-04T09:22:20.7470938Z 5b6f6de4bdd1: Waiting 2025-12-04T09:22:20.7471103Z 0d1243af7593: Waiting 2025-12-04T09:22:20.7471259Z 0aa3ea0b2754: Waiting 2025-12-04T09:22:20.7471419Z 4444f70d73ce: Waiting 2025-12-04T09:22:20.7471581Z cca4db1ba155: Waiting 2025-12-04T09:22:20.7471744Z 1f6d5b941ea1: Waiting 2025-12-04T09:22:20.7471904Z 821085416919: Waiting 2025-12-04T09:22:20.7472067Z f0ffa379f4eb: Waiting 2025-12-04T09:22:20.7472225Z c21a6e1cd03a: Waiting 2025-12-04T09:22:20.7472565Z 8c8d59b8759d: Waiting 2025-12-04T09:22:20.7472740Z 07dafc893dea: Waiting 2025-12-04T09:22:20.7472900Z 27803b661d9e: Waiting 2025-12-04T09:22:20.7473063Z 3c960865867b: Waiting 2025-12-04T09:22:20.7473231Z d1fba7293688: Waiting 2025-12-04T09:22:20.7473393Z 226748f4ff23: Waiting 2025-12-04T09:22:20.7473556Z 388169fffe8a: Waiting 2025-12-04T09:22:20.7473724Z ba3aee7dcf03: Waiting 2025-12-04T09:22:20.7473881Z df645e678f95: Waiting 2025-12-04T09:22:20.7474056Z 025a0e5e6ac1: Waiting 2025-12-04T09:22:20.7474211Z 6ca498e78b7b: Waiting 2025-12-04T09:22:20.7474360Z 75d817336f1d: Waiting 2025-12-04T09:22:20.7474515Z 74e0bdbb05d3: Waiting 2025-12-04T09:22:20.7474669Z 0f7d0dc70d2d: Waiting 2025-12-04T09:22:20.7474815Z aa9a10a37b39: Waiting 2025-12-04T09:22:20.7474969Z 3738646b3d92: Waiting 2025-12-04T09:22:20.7475126Z fa23d9952f3c: Waiting 2025-12-04T09:22:20.7475280Z bf45550ddbad: Waiting 2025-12-04T09:22:20.7475429Z 4f4fb700ef54: Waiting 2025-12-04T09:22:20.7475581Z acf6468f6aad: Waiting 2025-12-04T09:22:20.7475736Z 613d2b1f3db8: Waiting 2025-12-04T09:22:20.7475884Z 6a9b03ce41a7: Waiting 2025-12-04T09:22:20.7476041Z c2aa9ea4d09a: Waiting 2025-12-04T09:22:20.7476196Z b4e1efca22be: Waiting 2025-12-04T09:22:20.7476346Z c0c31a0e69b7: Waiting 2025-12-04T09:22:20.8250753Z 6dc15eca5138: Verifying Checksum 2025-12-04T09:22:20.8251073Z 6dc15eca5138: Download complete 2025-12-04T09:22:20.8967569Z 821085416919: Verifying Checksum 2025-12-04T09:22:20.8968045Z 821085416919: Download complete 2025-12-04T09:22:20.9640116Z 3738646b3d92: Download complete 2025-12-04T09:22:21.0578353Z aa9a10a37b39: Verifying Checksum 2025-12-04T09:22:21.0581081Z aa9a10a37b39: Download complete 2025-12-04T09:22:21.0930266Z 63e5bc7682b8: Verifying Checksum 2025-12-04T09:22:21.0930823Z 63e5bc7682b8: Download complete 2025-12-04T09:22:21.1329197Z 55bac08f3e18: Verifying Checksum 2025-12-04T09:22:21.1329718Z 55bac08f3e18: Download complete 2025-12-04T09:22:21.1715328Z acf6468f6aad: Verifying Checksum 2025-12-04T09:22:21.1718294Z acf6468f6aad: Download complete 2025-12-04T09:22:21.2554080Z 0f7d0dc70d2d: Download complete 2025-12-04T09:22:21.3295028Z 58c4d15d3bc1: Verifying Checksum 2025-12-04T09:22:21.3295320Z 58c4d15d3bc1: Download complete 2025-12-04T09:22:21.4270619Z 2f1cc47b61e1: Verifying Checksum 2025-12-04T09:22:21.4270945Z 2f1cc47b61e1: Download complete 2025-12-04T09:22:21.5054644Z d2472a74103d: Verifying Checksum 2025-12-04T09:22:21.5054961Z d2472a74103d: Download complete 2025-12-04T09:22:22.2092368Z 63e5bc7682b8: Pull complete 2025-12-04T09:22:22.2349997Z 6dc15eca5138: Pull complete 2025-12-04T09:22:22.5994730Z c0c31a0e69b7: Verifying Checksum 2025-12-04T09:22:22.5995255Z c0c31a0e69b7: Download complete 2025-12-04T09:22:22.6084079Z 4f4fb700ef54: Verifying Checksum 2025-12-04T09:22:22.6086506Z 4f4fb700ef54: Download complete 2025-12-04T09:22:22.7292982Z 6a9b03ce41a7: Download complete 2025-12-04T09:22:22.8556874Z 3519ac15be79: Verifying Checksum 2025-12-04T09:22:22.8562438Z 3519ac15be79: Download complete 2025-12-04T09:22:22.9435175Z dabb51b819a0: Verifying Checksum 2025-12-04T09:22:22.9435750Z dabb51b819a0: Download complete 2025-12-04T09:22:23.0942651Z 8f36833a24d0: Verifying Checksum 2025-12-04T09:22:23.0943171Z 8f36833a24d0: Download complete 2025-12-04T09:22:23.1709979Z ab53c5a853a5: Verifying Checksum 2025-12-04T09:22:23.1710692Z ab53c5a853a5: Download complete 2025-12-04T09:22:23.2620583Z 025a0e5e6ac1: Download complete 2025-12-04T09:22:23.3319557Z ce3394c8f210: Verifying Checksum 2025-12-04T09:22:23.3320083Z ce3394c8f210: Download complete 2025-12-04T09:22:23.4264097Z a5c3888c3a0c: Verifying Checksum 2025-12-04T09:22:23.4264381Z a5c3888c3a0c: Download complete 2025-12-04T09:22:23.5208147Z ed902a3a4e3b: Verifying Checksum 2025-12-04T09:22:23.5208432Z ed902a3a4e3b: Download complete 2025-12-04T09:22:23.6015913Z b4e1efca22be: Verifying Checksum 2025-12-04T09:22:23.6016373Z b4e1efca22be: Download complete 2025-12-04T09:22:23.6825941Z 79fcfd297d9f: Download complete 2025-12-04T09:22:23.7871063Z 7bd89c134b49: Verifying Checksum 2025-12-04T09:22:23.7871352Z 7bd89c134b49: Download complete 2025-12-04T09:22:24.0660565Z 459f4df18f07: Verifying Checksum 2025-12-04T09:22:24.0661141Z 459f4df18f07: Download complete 2025-12-04T09:22:24.1277622Z dd1cdd87320d: Verifying Checksum 2025-12-04T09:22:24.1277984Z dd1cdd87320d: Download complete 2025-12-04T09:22:24.1993429Z c21a6e1cd03a: Download complete 2025-12-04T09:22:24.2767892Z 0aa3ea0b2754: Download complete 2025-12-04T09:22:24.3616805Z 74e0bdbb05d3: Download complete 2025-12-04T09:22:24.9418079Z ba3aee7dcf03: Verifying Checksum 2025-12-04T09:22:24.9422489Z ba3aee7dcf03: Download complete 2025-12-04T09:22:25.0300340Z 8c8d59b8759d: Verifying Checksum 2025-12-04T09:22:25.0300654Z 8c8d59b8759d: Download complete 2025-12-04T09:22:25.1261154Z 3c960865867b: Verifying Checksum 2025-12-04T09:22:25.1262015Z 3c960865867b: Download complete 2025-12-04T09:22:25.2492678Z 074db79e3832: Verifying Checksum 2025-12-04T09:22:25.2497494Z 074db79e3832: Download complete 2025-12-04T09:22:25.3243910Z 226748f4ff23: Verifying Checksum 2025-12-04T09:22:25.3249173Z 226748f4ff23: Download complete 2025-12-04T09:22:25.4168042Z 75d817336f1d: Download complete 2025-12-04T09:22:28.9052342Z b7d40b4fd1b9: Verifying Checksum 2025-12-04T09:22:28.9052657Z b7d40b4fd1b9: Download complete 2025-12-04T09:22:28.9914887Z e6225129924f: Verifying Checksum 2025-12-04T09:22:28.9917695Z e6225129924f: Download complete 2025-12-04T09:22:29.0784510Z 5cd6b9ec5c3c: Download complete 2025-12-04T09:22:29.2287970Z 613d2b1f3db8: Verifying Checksum 2025-12-04T09:22:29.2288330Z 613d2b1f3db8: Download complete 2025-12-04T09:22:33.2692351Z 388169fffe8a: Verifying Checksum 2025-12-04T09:22:33.2692661Z 388169fffe8a: Download complete 2025-12-04T09:22:34.8967463Z 459f4df18f07: Pull complete 2025-12-04T09:22:35.2430107Z 821085416919: Pull complete 2025-12-04T09:22:35.5671332Z 3738646b3d92: Pull complete 2025-12-04T09:22:35.9425773Z aa9a10a37b39: Pull complete 2025-12-04T09:22:36.2798336Z 55bac08f3e18: Pull complete 2025-12-04T09:22:36.5997626Z acf6468f6aad: Pull complete 2025-12-04T09:22:40.0550845Z c0c31a0e69b7: Pull complete 2025-12-04T09:22:40.3415297Z 0f7d0dc70d2d: Pull complete 2025-12-04T09:22:40.6006947Z 58c4d15d3bc1: Pull complete 2025-12-04T09:22:40.8641063Z 2f1cc47b61e1: Pull complete 2025-12-04T09:22:41.1448036Z d2472a74103d: Pull complete 2025-12-04T09:23:03.2812115Z d1fba7293688: Verifying Checksum 2025-12-04T09:23:03.2812537Z d1fba7293688: Download complete 2025-12-04T09:23:03.3638157Z 952b753ec7be: Verifying Checksum 2025-12-04T09:23:03.3638463Z 952b753ec7be: Download complete 2025-12-04T09:23:03.4437479Z 5b6f6de4bdd1: Download complete 2025-12-04T09:23:03.5567218Z fa23d9952f3c: Verifying Checksum 2025-12-04T09:23:03.5567727Z fa23d9952f3c: Download complete 2025-12-04T09:23:03.6325456Z c2aa9ea4d09a: Verifying Checksum 2025-12-04T09:23:03.6329730Z c2aa9ea4d09a: Download complete 2025-12-04T09:23:03.7135391Z a9bbc9e426d3: Verifying Checksum 2025-12-04T09:23:03.7135940Z a9bbc9e426d3: Download complete 2025-12-04T09:23:03.7955405Z 82f2371f6dc2: Verifying Checksum 2025-12-04T09:23:03.7955718Z 82f2371f6dc2: Download complete 2025-12-04T09:23:03.8830020Z c87301921afa: Verifying Checksum 2025-12-04T09:23:03.8840091Z c87301921afa: Download complete 2025-12-04T09:23:03.9891879Z 384d566a822f: Verifying Checksum 2025-12-04T09:23:03.9892256Z 384d566a822f: Download complete 2025-12-04T09:23:04.1199885Z 32ddcdd5dd69: Verifying Checksum 2025-12-04T09:23:04.1200444Z 32ddcdd5dd69: Download complete 2025-12-04T09:23:04.2134354Z a116a32ceaa1: Verifying Checksum 2025-12-04T09:23:04.2134711Z a116a32ceaa1: Download complete 2025-12-04T09:23:04.2835916Z f6fbbbe6067e: Verifying Checksum 2025-12-04T09:23:04.2836213Z f6fbbbe6067e: Download complete 2025-12-04T09:23:04.3975743Z 648018658875: Download complete 2025-12-04T09:23:04.4610630Z 5e2f7eac20ad: Verifying Checksum 2025-12-04T09:23:04.4610947Z 5e2f7eac20ad: Download complete 2025-12-04T09:23:04.5663362Z 0d1243af7593: Verifying Checksum 2025-12-04T09:23:04.5663990Z 0d1243af7593: Download complete 2025-12-04T09:23:04.6603544Z 1f6d5b941ea1: Verifying Checksum 2025-12-04T09:23:04.6606837Z 1f6d5b941ea1: Download complete 2025-12-04T09:23:04.7493590Z 4444f70d73ce: Download complete 2025-12-04T09:23:04.8259823Z bf45550ddbad: Verifying Checksum 2025-12-04T09:23:04.8260134Z bf45550ddbad: Download complete 2025-12-04T09:23:08.2664434Z 30fff4f4bad3: Verifying Checksum 2025-12-04T09:23:08.2667464Z 30fff4f4bad3: Download complete 2025-12-04T09:23:08.3250380Z cca4db1ba155: Verifying Checksum 2025-12-04T09:23:08.3250666Z cca4db1ba155: Download complete 2025-12-04T09:23:08.4181429Z f0ffa379f4eb: Verifying Checksum 2025-12-04T09:23:08.4181967Z f0ffa379f4eb: Download complete 2025-12-04T09:23:08.4994566Z df645e678f95: Download complete 2025-12-04T09:23:08.5758318Z 52ad6c17d103: Verifying Checksum 2025-12-04T09:23:08.5758619Z 52ad6c17d103: Download complete 2025-12-04T09:23:08.6754417Z 34408374e32c: Download complete 2025-12-04T09:23:08.7388336Z 27803b661d9e: Download complete 2025-12-04T09:23:09.5383065Z 07dafc893dea: Verifying Checksum 2025-12-04T09:23:09.5383547Z 07dafc893dea: Download complete 2025-12-04T09:23:58.6008272Z 6ca498e78b7b: Download complete 2025-12-04T09:24:14.3586912Z d1fba7293688: Pull complete 2025-12-04T09:24:14.6147435Z 4f4fb700ef54: Pull complete 2025-12-04T09:24:14.6634378Z 6a9b03ce41a7: Pull complete 2025-12-04T09:24:14.7402077Z 3519ac15be79: Pull complete 2025-12-04T09:24:14.7842395Z dabb51b819a0: Pull complete 2025-12-04T09:24:14.8842189Z 8f36833a24d0: Pull complete 2025-12-04T09:24:14.9324342Z ab53c5a853a5: Pull complete 2025-12-04T09:24:14.9767305Z 025a0e5e6ac1: Pull complete 2025-12-04T09:24:15.1903584Z ce3394c8f210: Pull complete 2025-12-04T09:24:15.4547709Z a5c3888c3a0c: Pull complete 2025-12-04T09:24:15.7862822Z ed902a3a4e3b: Pull complete 2025-12-04T09:24:16.0381998Z b4e1efca22be: Pull complete 2025-12-04T09:24:16.6140124Z 79fcfd297d9f: Pull complete 2025-12-04T09:24:17.0298435Z 7bd89c134b49: Pull complete 2025-12-04T09:24:28.3400906Z b7d40b4fd1b9: Pull complete 2025-12-04T09:24:28.5422912Z dd1cdd87320d: Pull complete 2025-12-04T09:24:28.9661181Z c21a6e1cd03a: Pull complete 2025-12-04T09:24:29.8082999Z 0aa3ea0b2754: Pull complete 2025-12-04T09:24:30.2279280Z 74e0bdbb05d3: Pull complete 2025-12-04T09:24:30.9447212Z ba3aee7dcf03: Pull complete 2025-12-04T09:24:31.3839228Z 8c8d59b8759d: Pull complete 2025-12-04T09:24:31.7474217Z 3c960865867b: Pull complete 2025-12-04T09:24:32.6375214Z 074db79e3832: Pull complete 2025-12-04T09:24:32.8958564Z 226748f4ff23: Pull complete 2025-12-04T09:24:33.3428395Z 75d817336f1d: Pull complete 2025-12-04T09:26:23.5203912Z 850870a17b94: Verifying Checksum 2025-12-04T09:26:23.5204194Z 850870a17b94: Download complete 2025-12-04T09:30:23.7402078Z 850870a17b94: Pull complete 2025-12-04T09:30:24.2132164Z e6225129924f: Pull complete 2025-12-04T09:30:24.6686466Z 5cd6b9ec5c3c: Pull complete 2025-12-04T09:30:25.5040929Z 613d2b1f3db8: Pull complete 2025-12-04T09:30:28.4521147Z 388169fffe8a: Pull complete 2025-12-04T09:32:51.3128736Z 6ca498e78b7b: Pull complete 2025-12-04T09:32:51.7524383Z 952b753ec7be: Pull complete 2025-12-04T09:32:52.1803573Z 5b6f6de4bdd1: Pull complete 2025-12-04T09:32:53.0133222Z fa23d9952f3c: Pull complete 2025-12-04T09:32:53.7464445Z c2aa9ea4d09a: Pull complete 2025-12-04T09:32:53.9288298Z a9bbc9e426d3: Pull complete 2025-12-04T09:32:54.7128680Z 82f2371f6dc2: Pull complete 2025-12-04T09:32:55.4938264Z c87301921afa: Pull complete 2025-12-04T09:32:55.8479349Z 384d566a822f: Pull complete 2025-12-04T09:32:56.6826903Z 32ddcdd5dd69: Pull complete 2025-12-04T09:32:57.5636598Z a116a32ceaa1: Pull complete 2025-12-04T09:32:57.7931368Z f6fbbbe6067e: Pull complete 2025-12-04T09:32:58.6267541Z 648018658875: Pull complete 2025-12-04T09:32:58.9124965Z 5e2f7eac20ad: Pull complete 2025-12-04T09:32:59.7195496Z 0d1243af7593: Pull complete 2025-12-04T09:33:00.1997470Z 1f6d5b941ea1: Pull complete 2025-12-04T09:33:00.9672909Z 4444f70d73ce: Pull complete 2025-12-04T09:33:01.2887957Z bf45550ddbad: Pull complete 2025-12-04T09:33:09.7196153Z 30fff4f4bad3: Pull complete 2025-12-04T09:33:10.2488609Z cca4db1ba155: Pull complete 2025-12-04T09:33:10.6780776Z f0ffa379f4eb: Pull complete 2025-12-04T09:33:11.1642199Z df645e678f95: Pull complete 2025-12-04T09:33:11.5919756Z 52ad6c17d103: Pull complete 2025-12-04T09:33:12.1208407Z 34408374e32c: Pull complete 2025-12-04T09:33:13.1084823Z 27803b661d9e: Pull complete 2025-12-04T09:33:15.7184284Z 07dafc893dea: Pull complete 2025-12-04T09:33:16.4450034Z Digest: sha256:b178ee928adfcab963ebbc6ea05ea1a0f0c605bb095e9076f010bf92d150869d 2025-12-04T09:33:16.5529192Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:33:16.6042948Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:33:16.6092450Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:33:16.6093086Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:33:16.6101176Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:16.6101421Z env: 2025-12-04T09:33:16.6101579Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:16.6101752Z ##[endgroup] 2025-12-04T09:33:16.6238121Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-12-04T09:33:16.6238420Z with: 2025-12-04T09:33:16.6238594Z driver-version: 580.82.07 2025-12-04T09:33:16.6238776Z env: 2025-12-04T09:33:16.6238938Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:16.6239131Z ##[endgroup] 2025-12-04T09:33:16.6415840Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:33:16.6416620Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:33:16.6421251Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:16.6421491Z env: 2025-12-04T09:33:16.6421649Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:16.6421821Z ##[endgroup] 2025-12-04T09:33:16.6576540Z ##[group]Run set -euo pipefail 2025-12-04T09:33:16.6576797Z set -euo pipefail 2025-12-04T09:33:16.6577034Z  2025-12-04T09:33:16.6577198Z has_gpu=false 2025-12-04T09:33:16.6577388Z devices="" 2025-12-04T09:33:16.6577565Z  2025-12-04T09:33:16.6577765Z if command -v nvidia-smi >/dev/null 2>&1; then 2025-12-04T09:33:16.6578062Z  if nvidia-smi -L >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:33:16.6578318Z  has_gpu=true 2025-12-04T09:33:16.6578527Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:33:16.6578749Z  fi 2025-12-04T09:33:16.6578916Z fi 2025-12-04T09:33:16.6579079Z  2025-12-04T09:33:16.6579243Z if [ "$has_gpu" = false ]; then 2025-12-04T09:33:16.6579501Z  if ls /dev/nvidia* >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:33:16.6579751Z  has_gpu=true 2025-12-04T09:33:16.6579954Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:33:16.6580164Z  fi 2025-12-04T09:33:16.6580329Z fi 2025-12-04T09:33:16.6580494Z  2025-12-04T09:33:16.6580706Z if [ "$has_gpu" = false ] && command -v lspci >/dev/null 2>&1; then 2025-12-04T09:33:16.6581020Z  if lspci | grep -i 'nvidia' >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:33:16.6581287Z  has_gpu=true 2025-12-04T09:33:16.6581481Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:33:16.6581694Z  fi 2025-12-04T09:33:16.6581862Z fi 2025-12-04T09:33:16.6582012Z  2025-12-04T09:33:16.6582223Z printf 'HAS_NVIDIA=%s\n' "$has_gpu" >> "$GITHUB_OUTPUT" 2025-12-04T09:33:16.6582551Z printf 'DETECTED_DEVICES<> "$GITHUB_OUTPUT" 2025-12-04T09:33:16.6586918Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:16.6587149Z env: 2025-12-04T09:33:16.6587314Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:16.6587498Z ##[endgroup] 2025-12-04T09:33:16.8502998Z ##[group]Run if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T09:33:16.8503326Z if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T09:33:16.8503604Z  echo "HAS_NVIDIA_GPU=true" >> "${GITHUB_ENV}" 2025-12-04T09:33:16.8503955Z  echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2025-12-04T09:33:16.8504304Z else 2025-12-04T09:33:16.8504507Z  echo "HAS_NVIDIA_GPU=false" >> "${GITHUB_ENV}" 2025-12-04T09:33:16.8504760Z fi 2025-12-04T09:33:16.8509594Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:16.8509860Z env: 2025-12-04T09:33:16.8510035Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:16.8510235Z HAS_NVIDIA: false 2025-12-04T09:33:16.8510412Z ##[endgroup] 2025-12-04T09:33:16.8578740Z Prepare all required actions 2025-12-04T09:33:16.8622643Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-12-04T09:33:16.8622917Z with: 2025-12-04T09:33:16.8623556Z github-token: *** 2025-12-04T09:33:16.8623748Z env: 2025-12-04T09:33:16.8623923Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:16.8624144Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:16.8624359Z ##[endgroup] 2025-12-04T09:33:16.8680310Z ##[group]Run set -eux 2025-12-04T09:33:16.8680500Z set -eux 2025-12-04T09:33:16.8680785Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-12-04T09:33:16.8685536Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:16.8685923Z env: 2025-12-04T09:33:16.8686085Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:16.8686278Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:16.8686601Z GITHUB_TOKEN: *** 2025-12-04T09:33:16.8686761Z ##[endgroup] 2025-12-04T09:33:16.8709992Z + python3 .github/scripts/get_workflow_job_id.py 19923066595 i-0cbeb234d1c75d5ac 2025-12-04T09:33:19.1128186Z Setting output job-id=57118563268 2025-12-04T09:33:19.1132081Z Setting output job-name=periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:33:19.1236973Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T09:33:19.1237423Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T09:33:19.1237982Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-12-04T09:33:19.1238468Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:33:19.1243320Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:19.1243548Z env: 2025-12-04T09:33:19.1243704Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:19.1243882Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:19.1244042Z JOB_ID: 57118563268 2025-12-04T09:33:19.1244434Z JOB_NAME: periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:33:19.1244799Z WORKFLOW_NAME: inductor-periodic 2025-12-04T09:33:19.1245033Z WORKFLOW_RUN_ID: 19923066595 2025-12-04T09:33:19.1245209Z MONITOR_LOG_INTERVAL: 5 2025-12-04T09:33:19.1245392Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-12-04T09:33:19.1245578Z ##[endgroup] 2025-12-04T09:33:19.7862258Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:33:20.0962398Z Collecting psutil==5.9.8 2025-12-04T09:33:20.1129341Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-12-04T09:33:20.2949100Z Collecting dataclasses_json==0.6.7 2025-12-04T09:33:20.2988040Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-12-04T09:33:20.3528081Z Collecting nvidia-ml-py==11.525.84 2025-12-04T09:33:20.3568420Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-12-04T09:33:20.4485664Z Collecting typing-inspect<1,>=0.4.0 2025-12-04T09:33:20.4526683Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-12-04T09:33:20.6364027Z Collecting marshmallow<4.0.0,>=3.18.0 2025-12-04T09:33:20.6408866Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-12-04T09:33:20.7875564Z Collecting packaging>=17.0 2025-12-04T09:33:20.7913617Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-12-04T09:33:20.9095741Z Collecting typing-extensions>=3.7.4 2025-12-04T09:33:20.9138522Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-12-04T09:33:20.9980700Z Collecting mypy-extensions>=0.3.0 2025-12-04T09:33:21.0019413Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-12-04T09:33:21.2194883Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-12-04T09:33:22.1104959Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-12-04T09:33:22.3556094Z Prepare all required actions 2025-12-04T09:33:22.3556400Z Getting action download info 2025-12-04T09:33:22.5310866Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T09:33:23.8105709Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-12-04T09:33:24.7811411Z ##[group]Run ./.github/actions/download-build-artifacts 2025-12-04T09:33:24.7811823Z with: 2025-12-04T09:33:24.7812019Z name: linux-jammy-py3.10-gcc11-build 2025-12-04T09:33:24.7812441Z s3-bucket: gha-artifacts 2025-12-04T09:33:24.7812643Z env: 2025-12-04T09:33:24.7812805Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:24.7813015Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:24.7813210Z ##[endgroup] 2025-12-04T09:33:24.7834976Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T09:33:24.7835203Z with: 2025-12-04T09:33:24.7835376Z name: linux-jammy-py3.10-gcc11-build 2025-12-04T09:33:24.7835627Z s3-bucket: gha-artifacts 2025-12-04T09:33:24.7835802Z region: us-east-1 2025-12-04T09:33:24.7835945Z env: 2025-12-04T09:33:24.7836087Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:24.7836256Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:24.7836411Z ##[endgroup] 2025-12-04T09:33:25.1512089Z (node:52517) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T09:33:25.1512604Z 2025-12-04T09:33:25.1512846Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T09:33:25.1513250Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T09:33:25.1513588Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T09:33:25.4740039Z Found 1 objects with prefix pytorch/pytorch/19923066595/linux-jammy-py3.10-gcc11-build/ 2025-12-04T09:33:25.4740533Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T09:33:30.1956360Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T09:33:30.1961598Z Artifact download has finished successfully 2025-12-04T09:33:30.2134945Z ##[group]Run unzip -o artifacts.zip 2025-12-04T09:33:30.2135198Z unzip -o artifacts.zip 2025-12-04T09:33:30.2139930Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:30.2140166Z env: 2025-12-04T09:33:30.2140321Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:30.2140498Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:30.2140665Z ##[endgroup] 2025-12-04T09:33:30.2212276Z Archive: artifacts.zip 2025-12-04T09:33:30.2216122Z creating: dist/ 2025-12-04T09:33:31.3008490Z inflating: dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T09:33:31.3009752Z creating: dist/vision/ 2025-12-04T09:33:31.3080763Z inflating: dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:33:31.3082582Z creating: dist/audio/ 2025-12-04T09:33:31.3103009Z inflating: dist/audio/torchaudio-2.10.0a0+e90a398-cp310-cp310-linux_x86_64.whl 2025-12-04T09:33:31.3103472Z creating: dist/ao/ 2025-12-04T09:33:31.3138757Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-12-04T09:33:31.3251946Z inflating: dist/.ninja_log 2025-12-04T09:33:31.3252410Z creating: build/custom_test_artifacts/ 2025-12-04T09:33:31.3252829Z creating: build/custom_test_artifacts/custom-op-build/ 2025-12-04T09:33:31.3253193Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-12-04T09:33:31.3253627Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:33:31.3254204Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:33:31.3254703Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/ 2025-12-04T09:33:31.3255105Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:33:31.3255888Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:33:31.3256306Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:33:31.3257835Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:33:31.3258871Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:33:31.3259326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:33:31.3259880Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:33:31.3260382Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:33:31.3264717Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:33:31.3269496Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:33:31.3271684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:33:31.3272193Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:33:31.3272912Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:33:31.3273361Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:33:31.3273756Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-12-04T09:33:31.3274139Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-12-04T09:33:31.3274562Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-12-04T09:33:31.3275044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-12-04T09:33:31.3275497Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-12-04T09:33:31.3275919Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-12-04T09:33:31.3276350Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-12-04T09:33:31.3276788Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-12-04T09:33:31.3277223Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-12-04T09:33:31.3277661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-12-04T09:33:31.3278081Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-12-04T09:33:31.3289827Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-12-04T09:33:31.3464451Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-12-04T09:33:31.3465288Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-12-04T09:33:31.3465843Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-12-04T09:33:31.3466484Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-12-04T09:33:31.3467043Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-12-04T09:33:31.3467536Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-12-04T09:33:31.3468061Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-12-04T09:33:31.3468970Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-12-04T09:33:31.3469473Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-12-04T09:33:31.3469964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-12-04T09:33:31.3470473Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-12-04T09:33:31.3484477Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-12-04T09:33:31.3561768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-12-04T09:33:31.3562561Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:33:31.3563120Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:33:31.3563659Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-12-04T09:33:31.3564056Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-12-04T09:33:31.3564437Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-12-04T09:33:31.3564809Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-12-04T09:33:31.3565145Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-12-04T09:33:31.3565502Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-12-04T09:33:31.3720883Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-12-04T09:33:31.3770190Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-12-04T09:33:31.3770712Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-12-04T09:33:31.3771166Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-12-04T09:33:31.3771790Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:33:31.3772250Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:33:31.3778743Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/ 2025-12-04T09:33:31.3782686Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:33:31.3786722Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:33:31.3787218Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:33:31.3787671Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:33:31.3788121Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:33:31.3788539Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:33:31.3788935Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:33:31.3789322Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:33:31.3789779Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:33:31.3790252Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:33:31.3790687Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:33:31.3791156Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:33:31.3791659Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:33:31.3792398Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:33:31.3792790Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-12-04T09:33:31.3793168Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-12-04T09:33:31.3793583Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-12-04T09:33:31.3794072Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-12-04T09:33:31.3794645Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-12-04T09:33:31.3795083Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-12-04T09:33:31.3795520Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-12-04T09:33:31.3795982Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-12-04T09:33:31.3796437Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-12-04T09:33:31.3796892Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-12-04T09:33:31.3797328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-12-04T09:33:31.3804877Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-12-04T09:33:31.3858805Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-12-04T09:33:31.3859557Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:33:31.3860526Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:33:31.3865467Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-12-04T09:33:31.3867014Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-12-04T09:33:31.3867511Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-12-04T09:33:31.3873122Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-12-04T09:33:31.3875595Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-12-04T09:33:31.3876077Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-12-04T09:33:31.3898443Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-12-04T09:33:31.3903570Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-12-04T09:33:31.3905403Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-12-04T09:33:31.3905922Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:33:31.3910703Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:33:31.3911323Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/ 2025-12-04T09:33:31.3911800Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:33:31.3916447Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:33:31.3918258Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:33:31.3918811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:33:31.3919298Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:33:31.3919744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:33:31.3920399Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:33:31.3920835Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:33:31.3921366Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:33:31.3921856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:33:31.3922308Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:33:31.3922914Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:33:31.3923442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:33:31.3923901Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:33:31.3924303Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-12-04T09:33:31.3924716Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-12-04T09:33:31.3925163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-12-04T09:33:31.3925676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-12-04T09:33:31.3926183Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-12-04T09:33:31.3926647Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-12-04T09:33:31.3927125Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-12-04T09:33:31.3927612Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-12-04T09:33:31.3928090Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-12-04T09:33:31.3928568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-12-04T09:33:31.3929033Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-12-04T09:33:31.3929538Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-12-04T09:33:31.4019608Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-12-04T09:33:31.4024756Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-12-04T09:33:31.4029801Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-12-04T09:33:31.4031752Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-12-04T09:33:31.4032422Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-12-04T09:33:31.4036816Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-12-04T09:33:31.4042099Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-12-04T09:33:31.4046361Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-12-04T09:33:31.4046921Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-12-04T09:33:31.4047433Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-12-04T09:33:31.4048150Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-12-04T09:33:31.4048707Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-12-04T09:33:31.4086597Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-12-04T09:33:31.4090750Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:33:31.4093436Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:33:31.4094077Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-12-04T09:33:31.4099662Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-12-04T09:33:31.4102378Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-12-04T09:33:31.4102810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-12-04T09:33:31.4103157Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-12-04T09:33:31.4103518Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-12-04T09:33:31.4180997Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-12-04T09:33:31.4214470Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-12-04T09:33:31.4218441Z creating: build/lib/ 2025-12-04T09:33:31.4287523Z inflating: build/lib/libprotobuf-lite.a 2025-12-04T09:33:31.4678610Z inflating: build/lib/libprotobuf.a 2025-12-04T09:33:31.5119139Z inflating: build/lib/libprotoc.a 2025-12-04T09:33:31.5128666Z inflating: build/lib/libpthreadpool.a 2025-12-04T09:33:31.5135349Z inflating: build/lib/libcpuinfo.a 2025-12-04T09:33:31.5141544Z inflating: build/lib/libcpuinfo_internals.a 2025-12-04T09:33:31.5141813Z inflating: build/lib/libclog.a 2025-12-04T09:33:31.5160271Z inflating: build/lib/libpytorch_qnnpack.a 2025-12-04T09:33:31.5162445Z inflating: build/lib/libnnpack_reference_layers.a 2025-12-04T09:33:31.5323247Z inflating: build/lib/libmicrokernels-prod.a 2025-12-04T09:33:31.5339077Z inflating: build/lib/libnnpack.a 2025-12-04T09:33:31.6104147Z inflating: build/lib/libmicrokernels-all.a 2025-12-04T09:33:31.6164832Z inflating: build/lib/libgtest.a 2025-12-04T09:33:31.6179042Z inflating: build/lib/libgmock.a 2025-12-04T09:33:31.6180763Z inflating: build/lib/libgtest_main.a 2025-12-04T09:33:31.6181120Z inflating: build/lib/libgmock_main.a 2025-12-04T09:33:31.6256671Z inflating: build/lib/libXNNPACK.a 2025-12-04T09:33:31.6323238Z inflating: build/lib/libbenchmark.a 2025-12-04T09:33:31.6325373Z inflating: build/lib/libbenchmark_main.a 2025-12-04T09:33:31.6325822Z inflating: build/lib/libjitprofiling.a 2025-12-04T09:33:31.6330468Z inflating: build/lib/libittnotify.a 2025-12-04T09:33:31.6389141Z inflating: build/lib/libasmjit.a 2025-12-04T09:33:31.7392218Z inflating: build/lib/libfbgemm.a 2025-12-04T09:33:31.7416342Z inflating: build/lib/libtensorpipe_uv.a 2025-12-04T09:33:31.7888096Z inflating: build/lib/libtensorpipe.a 2025-12-04T09:33:31.7993763Z inflating: build/lib/libgloo.a 2025-12-04T09:33:31.8036970Z inflating: build/lib/libonnx_proto.a 2025-12-04T09:33:31.8667783Z inflating: build/lib/libonnx.a 2025-12-04T09:33:32.7451423Z inflating: build/lib/libdnnl.a 2025-12-04T09:33:32.7468982Z inflating: build/lib/libfmt.a 2025-12-04T09:33:32.7716560Z inflating: build/lib/libkineto.a 2025-12-04T09:33:32.7816007Z inflating: build/lib/libc10.so 2025-12-04T09:33:32.7820301Z inflating: build/lib/libtorch_global_deps.so 2025-12-04T09:33:35.5417349Z inflating: build/lib/libtorch_cpu.so 2025-12-04T09:33:35.5418309Z inflating: build/lib/libtorch.so 2025-12-04T09:33:35.5480392Z inflating: build/lib/libtorchbind_test.so 2025-12-04T09:33:35.5498080Z inflating: build/lib/libjitbackend_test.so 2025-12-04T09:33:35.5519558Z inflating: build/lib/libbackend_with_compiler.so 2025-12-04T09:33:35.5541597Z inflating: build/lib/libaoti_custom_ops.so 2025-12-04T09:33:35.5546372Z inflating: build/lib/libshm.so 2025-12-04T09:33:35.7579152Z inflating: build/lib/libtorch_python.so 2025-12-04T09:33:35.7612889Z inflating: build/lib/libnnapi_backend.so 2025-12-04T09:33:35.7613481Z creating: build/bin/ 2025-12-04T09:33:35.7613681Z creating: build/bin/CMakeFiles/ 2025-12-04T09:33:35.7613916Z inflating: build/bin/cmake_install.cmake 2025-12-04T09:33:35.7614137Z inflating: build/bin/CTestTestfile.cmake 2025-12-04T09:33:35.8025475Z inflating: build/bin/protoc-3.13.0.0 2025-12-04T09:33:35.8437408Z inflating: build/bin/protoc 2025-12-04T09:33:35.8492656Z inflating: build/bin/c10_AllocatorConfig_test 2025-12-04T09:33:35.8548762Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-12-04T09:33:35.8599859Z inflating: build/bin/c10_DeviceGuard_test 2025-12-04T09:33:35.8653834Z inflating: build/bin/c10_Device_test 2025-12-04T09:33:35.8720422Z inflating: build/bin/c10_DispatchKeySet_test 2025-12-04T09:33:35.8775194Z inflating: build/bin/c10_Scalar_test 2025-12-04T09:33:35.8826846Z inflating: build/bin/c10_StreamGuard_test 2025-12-04T09:33:35.8887478Z inflating: build/bin/c10_SymInt_test 2025-12-04T09:33:35.8939562Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-12-04T09:33:35.8993842Z inflating: build/bin/c10_InlineStreamGuard_test 2025-12-04T09:33:35.9049097Z inflating: build/bin/c10_SizesAndStrides_test 2025-12-04T09:33:35.9117827Z inflating: build/bin/c10_cow_test 2025-12-04T09:33:35.9169331Z inflating: build/bin/c10_Bitset_test 2025-12-04T09:33:35.9216881Z inflating: build/bin/c10_ArrayRef_test 2025-12-04T09:33:35.9265196Z inflating: build/bin/c10_ConstexprCrc_test 2025-12-04T09:33:35.9313600Z inflating: build/bin/c10_DeadlockDetection_test 2025-12-04T09:33:35.9368811Z inflating: build/bin/c10_Enumerate_test 2025-12-04T09:33:35.9422146Z inflating: build/bin/c10_LeftRight_test 2025-12-04T09:33:35.9476361Z inflating: build/bin/c10_NetworkFlow_test 2025-12-04T09:33:35.9525562Z inflating: build/bin/c10_Half_test 2025-12-04T09:33:35.9577589Z inflating: build/bin/c10_IntrusiveList_test 2025-12-04T09:33:35.9626484Z inflating: build/bin/c10_Synchronized_test 2025-12-04T09:33:35.9684155Z inflating: build/bin/c10_ThreadLocal_test 2025-12-04T09:33:35.9736159Z inflating: build/bin/c10_Semaphore_test 2025-12-04T09:33:35.9791468Z inflating: build/bin/c10_bfloat16_test 2025-12-04T09:33:35.9842349Z inflating: build/bin/c10_accumulate_test 2025-12-04T09:33:35.9893037Z inflating: build/bin/c10_TypeIndex_test 2025-12-04T09:33:35.9946390Z inflating: build/bin/c10_exception_test 2025-12-04T09:33:35.9996625Z inflating: build/bin/c10_bit_cast_test 2025-12-04T09:33:36.0049415Z inflating: build/bin/c10_complex_math_test 2025-12-04T09:33:36.0100279Z inflating: build/bin/c10_error_test 2025-12-04T09:33:36.0150371Z inflating: build/bin/c10_flags_test 2025-12-04T09:33:36.0203503Z inflating: build/bin/c10_complex_test 2025-12-04T09:33:36.0252165Z inflating: build/bin/c10_irange_test 2025-12-04T09:33:36.0302154Z inflating: build/bin/c10_generic_math_test 2025-12-04T09:33:36.0353898Z inflating: build/bin/c10_nofatal_test 2025-12-04T09:33:36.0405804Z inflating: build/bin/c10_lazy_test 2025-12-04T09:33:36.0462768Z inflating: build/bin/c10_logging_test 2025-12-04T09:33:36.0606166Z inflating: build/bin/c10_intrusive_ptr_test 2025-12-04T09:33:36.0678874Z inflating: build/bin/c10_optional_test 2025-12-04T09:33:36.0729232Z inflating: build/bin/c10_registry_test 2025-12-04T09:33:36.0791014Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-12-04T09:33:36.0841211Z inflating: build/bin/c10_ssize_test 2025-12-04T09:33:36.0985299Z inflating: build/bin/c10_small_vector_test 2025-12-04T09:33:36.1041307Z inflating: build/bin/c10_string_util_test 2025-12-04T09:33:36.1091797Z inflating: build/bin/c10_string_view_test 2025-12-04T09:33:36.1144003Z inflating: build/bin/c10_tempfile_test 2025-12-04T09:33:36.1186886Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-12-04T09:33:36.1245488Z inflating: build/bin/c10_typeid_test 2025-12-04T09:33:36.1779496Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-12-04T09:33:36.2324144Z inflating: build/bin/vec_test_all_types_AVX512 2025-12-04T09:33:36.2880852Z inflating: build/bin/vec_test_all_types_AVX2 2025-12-04T09:33:36.2972169Z inflating: build/bin/test_aoti_abi_check 2025-12-04T09:33:36.3024580Z inflating: build/bin/test_vec_half_DEFAULT 2025-12-04T09:33:36.3074870Z inflating: build/bin/test_vec_half_AVX512 2025-12-04T09:33:36.3126087Z inflating: build/bin/test_vec_half_AVX2 2025-12-04T09:33:36.3178133Z inflating: build/bin/BackoffTest 2025-12-04T09:33:36.3233065Z inflating: build/bin/FileStoreTest 2025-12-04T09:33:36.3286831Z inflating: build/bin/static_runtime_bench 2025-12-04T09:33:36.3529478Z inflating: build/bin/static_runtime_test 2025-12-04T09:33:36.3608116Z inflating: build/bin/Dict_test 2025-12-04T09:33:36.3666371Z inflating: build/bin/Dimname_test 2025-12-04T09:33:36.3728941Z inflating: build/bin/MaybeOwned_test 2025-12-04T09:33:36.3786847Z inflating: build/bin/NamedTensor_test 2025-12-04T09:33:36.3844136Z inflating: build/bin/apply_utils_test 2025-12-04T09:33:36.3905215Z inflating: build/bin/atest 2025-12-04T09:33:36.3968070Z inflating: build/bin/basic 2025-12-04T09:33:36.4023008Z inflating: build/bin/broadcast_test 2025-12-04T09:33:36.4072441Z inflating: build/bin/cpu_allocator_test 2025-12-04T09:33:36.4131585Z inflating: build/bin/cpu_generator_test 2025-12-04T09:33:36.4186724Z inflating: build/bin/cpu_profiling_allocator_test 2025-12-04T09:33:36.4276873Z inflating: build/bin/cpu_rng_test 2025-12-04T09:33:36.4327477Z inflating: build/bin/dlconvertor_test 2025-12-04T09:33:36.4385666Z inflating: build/bin/extension_backend_test 2025-12-04T09:33:36.4441263Z inflating: build/bin/half_test 2025-12-04T09:33:36.4536379Z inflating: build/bin/ivalue_test 2025-12-04T09:33:36.4582689Z inflating: build/bin/lazy_tensor_test 2025-12-04T09:33:36.4638230Z inflating: build/bin/math_kernel_test 2025-12-04T09:33:36.4694239Z inflating: build/bin/memory_format_test 2025-12-04T09:33:36.4754298Z inflating: build/bin/memory_overlapping_test 2025-12-04T09:33:36.4804337Z inflating: build/bin/mobile_memory_cleanup 2025-12-04T09:33:36.4861583Z inflating: build/bin/native_test 2025-12-04T09:33:36.4916488Z inflating: build/bin/operator_name_test 2025-12-04T09:33:36.4967833Z inflating: build/bin/operators_test 2025-12-04T09:33:36.5021829Z inflating: build/bin/packedtensoraccessor_test 2025-12-04T09:33:36.5090165Z inflating: build/bin/pow_test 2025-12-04T09:33:36.5147548Z inflating: build/bin/quantized_test 2025-12-04T09:33:36.5198068Z inflating: build/bin/reduce_ops_test 2025-12-04T09:33:36.5248402Z inflating: build/bin/reportMemoryUsage_test 2025-12-04T09:33:36.5308346Z inflating: build/bin/scalar_tensor_test 2025-12-04T09:33:36.5368060Z inflating: build/bin/scalar_test 2025-12-04T09:33:36.5418404Z inflating: build/bin/StorageUtils_test 2025-12-04T09:33:36.5473078Z inflating: build/bin/stride_properties_test 2025-12-04T09:33:36.5551340Z inflating: build/bin/tensor_iterator_test 2025-12-04T09:33:36.5630158Z inflating: build/bin/test_parallel 2025-12-04T09:33:36.5682416Z inflating: build/bin/thread_init_test 2025-12-04T09:33:36.5740281Z inflating: build/bin/type_ptr_test 2025-12-04T09:33:36.5796311Z inflating: build/bin/type_test 2025-12-04T09:33:36.5847456Z inflating: build/bin/undefined_tensor_test 2025-12-04T09:33:36.5899396Z inflating: build/bin/verify_api_visibility 2025-12-04T09:33:36.5971043Z inflating: build/bin/legacy_vmap_test 2025-12-04T09:33:36.6021980Z inflating: build/bin/weakref_test 2025-12-04T09:33:36.6073478Z inflating: build/bin/wrapdim_test 2025-12-04T09:33:36.6126523Z inflating: build/bin/xla_tensor_test 2025-12-04T09:33:36.6185206Z inflating: build/bin/IListRef_test 2025-12-04T09:33:36.6288698Z inflating: build/bin/List_test 2025-12-04T09:33:36.6353705Z inflating: build/bin/KernelFunction_test 2025-12-04T09:33:36.6468146Z inflating: build/bin/kernel_function_legacy_test 2025-12-04T09:33:36.6560053Z inflating: build/bin/kernel_function_test 2025-12-04T09:33:36.6684697Z inflating: build/bin/kernel_lambda_legacy_test 2025-12-04T09:33:36.6784321Z inflating: build/bin/kernel_lambda_test 2025-12-04T09:33:36.6842637Z inflating: build/bin/kernel_stackbased_test 2025-12-04T09:33:36.6939647Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-12-04T09:33:36.6994433Z inflating: build/bin/CppSignature_test 2025-12-04T09:33:36.7047703Z inflating: build/bin/backend_fallback_test 2025-12-04T09:33:36.7098299Z inflating: build/bin/op_allowlist_test 2025-12-04T09:33:36.7403200Z inflating: build/bin/op_registration_test 2025-12-04T09:33:36.7473151Z inflating: build/bin/inline_container_test 2025-12-04T09:33:36.7819320Z inflating: build/bin/test_lazy 2025-12-04T09:33:36.7879733Z inflating: build/bin/TCPStoreTest 2025-12-04T09:33:36.7933143Z inflating: build/bin/HashStoreTest 2025-12-04T09:33:36.9008892Z inflating: build/bin/test_jit 2025-12-04T09:33:36.9077227Z inflating: build/bin/ProcessGroupGlooTest 2025-12-04T09:33:36.9140734Z inflating: build/bin/test_aoti_inference 2025-12-04T09:33:36.9143261Z inflating: build/bin/example_allreduce 2025-12-04T09:33:36.9195126Z inflating: build/bin/test_dist_autograd 2025-12-04T09:33:36.9262778Z inflating: build/bin/test_cpp_rpc 2025-12-04T09:33:36.9267932Z inflating: build/bin/parallel_benchmark 2025-12-04T09:33:37.0397539Z inflating: build/bin/test_api 2025-12-04T09:33:37.0400081Z inflating: build/bin/torch_shm_manager 2025-12-04T09:33:37.0400481Z creating: .additional_ci_files/ 2025-12-04T09:33:37.0465000Z inflating: .additional_ci_files/test-times.json 2025-12-04T09:33:37.0697973Z inflating: .additional_ci_files/test-class-times.json 2025-12-04T09:33:37.0741744Z ##[group]Run rm artifacts.zip 2025-12-04T09:33:37.0741982Z rm artifacts.zip 2025-12-04T09:33:37.0747971Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:37.0748237Z env: 2025-12-04T09:33:37.0748398Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:37.0748598Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:37.0748787Z ##[endgroup] 2025-12-04T09:33:37.1403793Z ##[group]Run df -H 2025-12-04T09:33:37.1403981Z df -H 2025-12-04T09:33:37.1408408Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:37.1408669Z env: 2025-12-04T09:33:37.1408821Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:37.1409003Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:37.1409179Z ##[endgroup] 2025-12-04T09:33:37.1447912Z Filesystem Size Used Avail Use% Mounted on 2025-12-04T09:33:37.1448204Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-12-04T09:33:37.1448612Z tmpfs 67G 0 67G 0% /dev/shm 2025-12-04T09:33:37.1448943Z tmpfs 27G 791k 27G 1% /run 2025-12-04T09:33:37.1449149Z /dev/nvme0n1p1 215G 71G 144G 34% / 2025-12-04T09:33:37.1449434Z tmpfs 67G 13k 67G 1% /tmp 2025-12-04T09:33:37.1449673Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-12-04T09:33:37.1473088Z Prepare all required actions 2025-12-04T09:33:37.1473998Z Getting action download info 2025-12-04T09:33:37.4458341Z ##[group]Run ./.github/actions/download-td-artifacts 2025-12-04T09:33:37.4458645Z with: 2025-12-04T09:33:37.4458828Z env: 2025-12-04T09:33:37.4459006Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:37.4459232Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:37.4459439Z ##[endgroup] 2025-12-04T09:33:37.5101961Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T09:33:37.5102247Z with: 2025-12-04T09:33:37.5102405Z name: td_results 2025-12-04T09:33:37.5102578Z s3-bucket: gha-artifacts 2025-12-04T09:33:37.5102767Z region: us-east-1 2025-12-04T09:33:37.5102924Z env: 2025-12-04T09:33:37.5103081Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:37.5103264Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:37.5103439Z ##[endgroup] 2025-12-04T09:33:38.1431446Z (node:52539) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T09:33:38.1432016Z 2025-12-04T09:33:38.1432186Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T09:33:38.1432597Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T09:33:38.1433070Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T09:33:38.2414981Z Found 0 objects with prefix pytorch/pytorch/19923066595/td_results/ 2025-12-04T09:33:38.2419994Z Artifact download has finished successfully 2025-12-04T09:33:38.2583965Z ##[group]Run mkdir -p .additional_ci_files 2025-12-04T09:33:38.2584235Z mkdir -p .additional_ci_files 2025-12-04T09:33:38.2584525Z mv td_results.json .additional_ci_files/td_results.json || true 2025-12-04T09:33:38.2589320Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:38.2589561Z env: 2025-12-04T09:33:38.2589741Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:38.2589914Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:38.2590089Z ##[endgroup] 2025-12-04T09:33:38.2639576Z mv: cannot stat 'td_results.json': No such file or directory 2025-12-04T09:33:38.2672711Z ##[group]Run .github/scripts/parse_ref.py 2025-12-04T09:33:38.2673016Z .github/scripts/parse_ref.py 2025-12-04T09:33:38.2677458Z shell: /usr/bin/bash -e {0} 2025-12-04T09:33:38.2677659Z env: 2025-12-04T09:33:38.2677826Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:38.2678021Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:38.2678197Z ##[endgroup] 2025-12-04T09:33:38.2864387Z Setting output branch=main 2025-12-04T09:33:38.2956959Z Prepare all required actions 2025-12-04T09:33:38.2957292Z Getting action download info 2025-12-04T09:33:38.4411235Z ##[group]Run ./.github/actions/filter-test-configs 2025-12-04T09:33:38.4411478Z with: 2025-12-04T09:33:38.4411936Z github-token: *** 2025-12-04T09:33:38.4418243Z test-matrix: {"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]} 2025-12-04T09:33:38.4424914Z job-name: periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:33:38.4425290Z env: 2025-12-04T09:33:38.4425445Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:38.4425759Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:38.4425938Z ##[endgroup] 2025-12-04T09:33:38.4450968Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T09:33:38.4451184Z with: 2025-12-04T09:33:38.4451340Z shell: bash 2025-12-04T09:33:38.4451509Z timeout_minutes: 10 2025-12-04T09:33:38.4451830Z max_attempts: 5 2025-12-04T09:33:38.4452009Z retry_wait_seconds: 30 2025-12-04T09:33:38.4452535Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T09:33:38.4453054Z polling_interval_seconds: 1 2025-12-04T09:33:38.4453246Z warning_on_retry: true 2025-12-04T09:33:38.4453431Z continue_on_error: false 2025-12-04T09:33:38.4453612Z env: 2025-12-04T09:33:38.4453758Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:38.4453945Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:38.4454251Z GITHUB_TOKEN: *** 2025-12-04T09:33:38.4454419Z ##[endgroup] 2025-12-04T09:33:38.5328166Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T09:33:38.7101974Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:33:38.7958595Z Collecting requests==2.27.1 2025-12-04T09:33:38.8103972Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-12-04T09:33:38.9369942Z Collecting pyyaml==6.0.2 2025-12-04T09:33:38.9448554Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-12-04T09:33:38.9677458Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-12-04T09:33:38.9678309Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-12-04T09:33:39.2315085Z Collecting charset-normalizer~=2.0.0 2025-12-04T09:33:39.2355359Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-12-04T09:33:39.2763207Z Collecting certifi>=2017.4.17 2025-12-04T09:33:39.2806051Z Downloading certifi-2025.11.12-py3-none-any.whl (159 kB) 2025-12-04T09:33:39.3470885Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-12-04T09:33:39.4509543Z Successfully installed certifi-2025.11.12 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-12-04T09:33:40.5102670Z Command completed after 1 attempt(s). 2025-12-04T09:33:40.5154686Z ##[group]Run set -x 2025-12-04T09:33:40.5154880Z set -x 2025-12-04T09:33:40.5155036Z  2025-12-04T09:33:40.5155290Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T09:33:40.5155581Z # in runner workspace 2025-12-04T09:33:40.5155845Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-12-04T09:33:40.5161478Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:40.5161732Z env: 2025-12-04T09:33:40.5161890Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:40.5162087Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:40.5162294Z ##[endgroup] 2025-12-04T09:33:40.5186255Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-12-04T09:33:40.5333060Z Setting output branch=main 2025-12-04T09:33:40.5382669Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T09:33:40.5382977Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T09:33:40.5383248Z echo "Job name: ${JOB_NAME}" 2025-12-04T09:33:40.5383447Z  2025-12-04T09:33:40.5383707Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T09:33:40.5384010Z # in runner workspace 2025-12-04T09:33:40.5384301Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-12-04T09:33:40.5384617Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-12-04T09:33:40.5384846Z  --job-name "${JOB_NAME}" \ 2025-12-04T09:33:40.5391297Z  --test-matrix "{"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]}" \ 2025-12-04T09:33:40.5397920Z  --selected-test-configs "" \ 2025-12-04T09:33:40.5398211Z  --pr-number "${PR_NUMBER}" \ 2025-12-04T09:33:40.5398458Z  --tag "${TAG}" \ 2025-12-04T09:33:40.5398686Z  --event-name "${EVENT_NAME}" \ 2025-12-04T09:33:40.5398938Z  --schedule "${SCHEDULE}" \ 2025-12-04T09:33:40.5399182Z  --branch "${HEAD_BRANCH}" 2025-12-04T09:33:40.5404402Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:40.5404722Z env: 2025-12-04T09:33:40.5404914Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:40.5405182Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:40.5405613Z GITHUB_TOKEN: *** 2025-12-04T09:33:40.5406093Z JOB_NAME: periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:33:40.5406520Z PR_NUMBER: 2025-12-04T09:33:40.5406694Z TAG: 2025-12-04T09:33:40.5406869Z EVENT_NAME: schedule 2025-12-04T09:33:40.5407085Z SCHEDULE: 45 0,4,8,12,16,20 * * 1-5 2025-12-04T09:33:40.5407306Z HEAD_BRANCH: main 2025-12-04T09:33:40.5407500Z ##[endgroup] 2025-12-04T09:33:40.5431735Z Workflow: inductor-periodic 2025-12-04T09:33:40.5432169Z Job name: periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:33:40.7269806Z Setting output keep-going=True 2025-12-04T09:33:40.7271880Z Setting output ci-verbose-test-logs=False 2025-12-04T09:33:40.7272144Z Setting output ci-test-showlocals=False 2025-12-04T09:33:40.7272473Z Setting output ci-no-test-timeout=False 2025-12-04T09:33:40.7272716Z Setting output ci-no-td=False 2025-12-04T09:33:40.7272918Z Setting output ci-td-distributed=False 2025-12-04T09:33:40.7273117Z Setting output is-unstable=False 2025-12-04T09:33:40.7273315Z Setting output reenabled-issues= 2025-12-04T09:33:40.7279240Z Setting output test-matrix={"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]} 2025-12-04T09:33:40.7285236Z Setting output is-test-matrix-empty=False 2025-12-04T09:33:40.7400339Z ##[group]Run echo "Filtered matrix:" 2025-12-04T09:33:40.7400589Z echo "Filtered matrix:" 2025-12-04T09:33:40.7406325Z echo "{"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_aot_inductor_amp_freezing_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}]}" 2025-12-04T09:33:40.7412457Z  2025-12-04T09:33:40.7412658Z echo 2025-12-04T09:33:40.7412894Z echo "Is the current job unstable? False" 2025-12-04T09:33:40.7413170Z  2025-12-04T09:33:40.7413365Z echo 2025-12-04T09:33:40.7413593Z echo "Is keep-going label set? True" 2025-12-04T09:33:40.7413822Z  2025-12-04T09:33:40.7413994Z echo 2025-12-04T09:33:40.7414184Z echo "Reenabled issues? " 2025-12-04T09:33:40.7419600Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:40.7419907Z env: 2025-12-04T09:33:40.7420099Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:40.7420311Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:40.7420508Z ##[endgroup] 2025-12-04T09:33:40.7443927Z Filtered matrix: 2025-12-04T09:33:40.7450586Z {include: [{config: cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_avx2_huggingface, shard: 1, num_shards: 1, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_torchbench, shard: 1, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_torchbench, shard: 2, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_timm, shard: 1, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_timm, shard: 2, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_amp_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_amp_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_aot_inductor_amp_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_aot_inductor_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_aot_inductor_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_aot_inductor_amp_freezing_torchbench, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_aot_inductor_amp_freezing_torchbench, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}]} 2025-12-04T09:33:40.7456972Z 2025-12-04T09:33:40.7457085Z Is the current job unstable? False 2025-12-04T09:33:40.7457261Z 2025-12-04T09:33:40.7457364Z Is keep-going label set? True 2025-12-04T09:33:40.7457496Z 2025-12-04T09:33:40.7457567Z Reenabled issues? 2025-12-04T09:33:40.7557113Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T09:33:40.7557435Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T09:33:40.7562021Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:40.7562245Z env: 2025-12-04T09:33:40.7562393Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:40.7562571Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:40.7562727Z JOB_TIMEOUT: 240 2025-12-04T09:33:40.7562882Z ##[endgroup] 2025-12-04T09:33:40.7719538Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:33:40.7719866Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:33:40.7720144Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:33:40.7724013Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:33:40.7724255Z env: 2025-12-04T09:33:40.7724411Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:40.7724580Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:40.7724736Z ##[endgroup] 2025-12-04T09:33:40.7877932Z ##[group]Run set -x 2025-12-04T09:33:40.7878160Z set -x 2025-12-04T09:33:40.7878306Z  2025-12-04T09:33:40.7878476Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-12-04T09:33:40.7878711Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-12-04T09:33:40.7878946Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-12-04T09:33:40.7879167Z  TEST_COMMAND=.ci/onnx/test.sh 2025-12-04T09:33:40.7879344Z else 2025-12-04T09:33:40.7879505Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T09:33:40.7879690Z fi 2025-12-04T09:33:40.7879814Z  2025-12-04T09:33:40.7879979Z # Leaving 1GB for the runner and other things 2025-12-04T09:33:40.7880314Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-12-04T09:33:40.7880923Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-12-04T09:33:40.7881323Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-12-04T09:33:40.7881629Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-12-04T09:33:40.7881869Z  2025-12-04T09:33:40.7882035Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T09:33:40.7882244Z  SHM_OPTS= 2025-12-04T09:33:40.7882404Z  JENKINS_USER= 2025-12-04T09:33:40.7882610Z  # ensure that docker container cleanly exits in 12 hours 2025-12-04T09:33:40.7882883Z  # if for some reason cleanup action doesn't stop container 2025-12-04T09:33:40.7883112Z  # when job is cancelled 2025-12-04T09:33:40.7883305Z  DOCKER_SHELL_CMD="sleep 12h" 2025-12-04T09:33:40.7883504Z  USED_IMAGE="${DOCKER_IMAGE_S390X}" 2025-12-04T09:33:40.7883686Z else 2025-12-04T09:33:40.7883842Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-12-04T09:33:40.7884038Z  JENKINS_USER="--user jenkins" 2025-12-04T09:33:40.7884225Z  DOCKER_SHELL_CMD= 2025-12-04T09:33:40.7884399Z  USED_IMAGE="${DOCKER_IMAGE}" 2025-12-04T09:33:40.7884565Z fi 2025-12-04T09:33:40.7884693Z  2025-12-04T09:33:40.7884893Z # detached container should get cleaned up by teardown_ec2_linux 2025-12-04T09:33:40.7885191Z # TODO: Stop building test binaries as part of the build phase 2025-12-04T09:33:40.7885521Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-12-04T09:33:40.7885816Z # shellcheck disable=SC2086,SC2090 2025-12-04T09:33:40.7886018Z container_name=$(docker run \ 2025-12-04T09:33:40.7886198Z  ${GPU_FLAG:-} \ 2025-12-04T09:33:40.7886391Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-12-04T09:33:40.7886596Z  -e BUILD_ENVIRONMENT \ 2025-12-04T09:33:40.7886780Z  -e PR_NUMBER \ 2025-12-04T09:33:40.7886952Z  -e GITHUB_ACTIONS \ 2025-12-04T09:33:40.7887137Z  -e GITHUB_REPOSITORY \ 2025-12-04T09:33:40.7887322Z  -e GITHUB_WORKFLOW \ 2025-12-04T09:33:40.7887499Z  -e GITHUB_JOB \ 2025-12-04T09:33:40.7887670Z  -e GITHUB_RUN_ID \ 2025-12-04T09:33:40.7887849Z  -e GITHUB_RUN_NUMBER \ 2025-12-04T09:33:40.7888027Z  -e GITHUB_RUN_ATTEMPT \ 2025-12-04T09:33:40.7888211Z  -e JOB_ID \ 2025-12-04T09:33:40.7888374Z  -e JOB_NAME \ 2025-12-04T09:33:40.7888540Z  -e BASE_SHA \ 2025-12-04T09:33:40.7888694Z  -e BRANCH \ 2025-12-04T09:33:40.7888850Z  -e SHA1 \ 2025-12-04T09:33:40.7889010Z  -e AWS_DEFAULT_REGION \ 2025-12-04T09:33:40.7889184Z  -e IN_WHEEL_TEST \ 2025-12-04T09:33:40.7889359Z  -e SHARD_NUMBER \ 2025-12-04T09:33:40.7889530Z  -e TEST_CONFIG \ 2025-12-04T09:33:40.7889696Z  -e NUM_TEST_SHARDS \ 2025-12-04T09:33:40.7889953Z  -e REENABLED_ISSUES \ 2025-12-04T09:33:40.7890154Z  -e CONTINUE_THROUGH_ERROR \ 2025-12-04T09:33:40.7890339Z  -e VERBOSE_TEST_LOGS \ 2025-12-04T09:33:40.7890521Z  -e TEST_SHOWLOCALS \ 2025-12-04T09:33:40.7890697Z  -e NO_TEST_TIMEOUT \ 2025-12-04T09:33:40.7890872Z  -e NO_TD \ 2025-12-04T09:33:40.7891029Z  -e TD_DISTRIBUTED \ 2025-12-04T09:33:40.7891204Z  -e PR_LABELS \ 2025-12-04T09:33:40.7891404Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-12-04T09:33:40.7891810Z  -e SCCACHE_BUCKET \ 2025-12-04T09:33:40.7892020Z  -e SCCACHE_REGION \ 2025-12-04T09:33:40.7892212Z  -e XLA_CUDA \ 2025-12-04T09:33:40.7892411Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-12-04T09:33:40.7892716Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-12-04T09:33:40.7892969Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-12-04T09:33:40.7893251Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-12-04T09:33:40.7893455Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-12-04T09:33:40.7893658Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-12-04T09:33:40.7893864Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-12-04T09:33:40.7894049Z  -e DASHBOARD_TAG \ 2025-12-04T09:33:40.7894229Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-12-04T09:33:40.7894457Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-12-04T09:33:40.7894729Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-12-04T09:33:40.7894983Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-12-04T09:33:40.7895225Z  --security-opt seccomp=unconfined \ 2025-12-04T09:33:40.7895423Z  --cap-add=SYS_PTRACE \ 2025-12-04T09:33:40.7895607Z  --ipc=host \ 2025-12-04T09:33:40.7895778Z  ${SHM_OPTS} \ 2025-12-04T09:33:40.7895938Z  --tty \ 2025-12-04T09:33:40.7896082Z  --detach \ 2025-12-04T09:33:40.7896256Z  --name="${container_name}" \ 2025-12-04T09:33:40.7896449Z  ${JENKINS_USER} \ 2025-12-04T09:33:40.7896657Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-12-04T09:33:40.7896896Z  -w /var/lib/jenkins/workspace \ 2025-12-04T09:33:40.7897090Z  "${USED_IMAGE}" \ 2025-12-04T09:33:40.7897254Z  ${DOCKER_SHELL_CMD} 2025-12-04T09:33:40.7897422Z ) 2025-12-04T09:33:40.7897636Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-12-04T09:33:40.7897876Z  2025-12-04T09:33:40.7898042Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T09:33:40.7898384Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-12-04T09:33:40.7898686Z fi 2025-12-04T09:33:40.7898813Z  2025-12-04T09:33:40.7899103Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-12-04T09:33:40.7903394Z shell: /usr/bin/bash -e {0} 2025-12-04T09:33:40.7903562Z env: 2025-12-04T09:33:40.7903708Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:33:40.7903885Z HAS_NVIDIA_GPU: false 2025-12-04T09:33:40.7904092Z BUILD_ENVIRONMENT: linux-jammy-py3.10-gcc11-build 2025-12-04T09:33:40.7904301Z PR_NUMBER: 2025-12-04T09:33:40.7904459Z GITHUB_REPOSITORY: pytorch/pytorch 2025-12-04T09:33:40.7904670Z GITHUB_WORKFLOW: inductor-periodic 2025-12-04T09:33:40.7904843Z GITHUB_JOB: test 2025-12-04T09:33:40.7904995Z GITHUB_RUN_ID: 19923066595 2025-12-04T09:33:40.7905166Z GITHUB_RUN_NUMBER: 67027 2025-12-04T09:33:40.7905322Z GITHUB_RUN_ATTEMPT: 1 2025-12-04T09:33:40.7905479Z JOB_ID: 57118563268 2025-12-04T09:33:40.7905790Z JOB_NAME: periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:33:40.7906122Z BRANCH: main 2025-12-04T09:33:40.7906290Z SHA1: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:33:40.7906585Z BASE_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:33:40.7906807Z TEST_CONFIG: cpu_inductor_huggingface 2025-12-04T09:33:40.7906988Z SHARD_NUMBER: 1 2025-12-04T09:33:40.7907135Z NUM_TEST_SHARDS: 1 2025-12-04T09:33:40.7907283Z EXTRA_FLAGS: 2025-12-04T09:33:40.7907423Z OP_BENCHMARK_TESTS: 2025-12-04T09:33:40.7907578Z REENABLED_ISSUES: 2025-12-04T09:33:40.7907736Z CONTINUE_THROUGH_ERROR: True 2025-12-04T09:33:40.7907902Z VERBOSE_TEST_LOGS: False 2025-12-04T09:33:40.7908072Z TEST_SHOWLOCALS: False 2025-12-04T09:33:40.7908239Z NO_TEST_TIMEOUT: False 2025-12-04T09:33:40.7908384Z NO_TD: False 2025-12-04T09:33:40.7908526Z TD_DISTRIBUTED: False 2025-12-04T09:33:40.7908718Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-12-04T09:33:40.7908933Z SCCACHE_REGION: us-east-1 2025-12-04T09:33:40.7909139Z SHM_SIZE: 1g 2025-12-04T09:33:40.7909610Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:33:40.7910414Z DOCKER_IMAGE_S390X: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:33:40.7910898Z XLA_CUDA: 2025-12-04T09:33:40.7911111Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:33:40.7911377Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-12-04T09:33:40.7911573Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-12-04T09:33:40.7911743Z DASHBOARD_TAG: 2025-12-04T09:33:40.7912049Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-12-04T09:33:40.7912300Z HUGGING_FACE_HUB_TOKEN: *** 2025-12-04T09:33:40.7912561Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-12-04T09:33:40.7912850Z ARTIFACTS_FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268 2025-12-04T09:33:40.7913129Z ##[endgroup] 2025-12-04T09:33:40.7931790Z + [[ cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-12-04T09:33:40.7932237Z + [[ linux-jammy-py3.10-gcc11-build == *onnx* ]] 2025-12-04T09:33:40.7935124Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T09:33:40.7940444Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-12-04T09:33:40.7955938Z + TOTAL_AVAILABLE_MEMORY_IN_GB='122.780 ' 2025-12-04T09:33:40.7958543Z + TOTAL_MEMORY_WITH_SWAP=125 2025-12-04T09:33:40.7958921Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-12-04T09:33:40.7961606Z + SHM_OPTS=--shm-size=1g 2025-12-04T09:33:40.7962036Z + JENKINS_USER='--user jenkins' 2025-12-04T09:33:40.7962255Z + DOCKER_SHELL_CMD= 2025-12-04T09:33:40.7963237Z + USED_IMAGE=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:33:40.7966758Z +++ nproc --ignore=2 2025-12-04T09:33:40.8403692Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=122g --memory-swap=125g --env-file=/tmp/github_env_19923066595 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:33:52.9488528Z + container_name=b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:33:52.9489195Z + echo DOCKER_CONTAINER_ID=b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:33:52.9490991Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-12-04T09:33:52.9494820Z ++ echo dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T09:33:52.9497562Z + docker exec -t b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 sh -c 'python3 -m pip install dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-12-04T09:33:53.4114836Z Processing ./dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl (from torch==2.10.0a0+gitffd9b0f) 2025-12-04T09:33:53.6278424Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.18.0) 2025-12-04T09:33:53.6281408Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (4.15.0) 2025-12-04T09:33:53.6282367Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.13.3) 2025-12-04T09:33:53.6286094Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2.8.8) 2025-12-04T09:33:53.6286960Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.1.6) 2025-12-04T09:33:53.6290239Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2025.10.0) 2025-12-04T09:33:53.6297803Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.3.0) 2025-12-04T09:33:53.6589548Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.22.4) 2025-12-04T09:33:53.6607945Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.3.0) 2025-12-04T09:33:53.6650393Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.0.3) 2025-12-04T09:33:54.4853204Z Installing collected packages: torch 2025-12-04T09:34:01.8424248Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-12-04T09:34:01.8424896Z dall-e 0.1 requires torchvision, which is not installed. 2025-12-04T09:34:01.8430045Z effdet 0.4.1 requires torchvision, which is not installed. 2025-12-04T09:34:01.8435601Z python-doctr 1.0.0 requires torchvision>=0.15.0, which is not installed. 2025-12-04T09:34:01.8436050Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-12-04T09:34:01.8436581Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-12-04T09:34:01.8437142Z timm 1.0.22 requires torchvision, which is not installed. 2025-12-04T09:34:01.8437481Z Successfully installed torch-2.10.0a0+gitffd9b0f 2025-12-04T09:34:01.9588524Z + export TERM=vt100 2025-12-04T09:34:01.9591241Z + TERM=vt100 2025-12-04T09:34:01.9591611Z ++ dirname .ci/pytorch/test.sh 2025-12-04T09:34:01.9600912Z + source .ci/pytorch/common.sh 2025-12-04T09:34:01.9602741Z +++ dirname .ci/pytorch/common.sh 2025-12-04T09:34:01.9611953Z ++ source .ci/pytorch/common_utils.sh 2025-12-04T09:34:01.9612282Z +++ declare -f -t trap_add 2025-12-04T09:34:01.9612859Z ++ set -ex -o pipefail 2025-12-04T09:34:01.9613094Z ++ [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T09:34:01.9613352Z ++ BUILD_TEST_LIBTORCH=0 2025-12-04T09:34:01.9613546Z ++ dirname .ci/pytorch/test.sh 2025-12-04T09:34:01.9627558Z + source .ci/pytorch/common-build.sh 2025-12-04T09:34:01.9628053Z ++ [[ linux-jammy-py3.10-gcc11-build != *win-* ]] 2025-12-04T09:34:01.9636119Z ++++ dirname .ci/pytorch/common-build.sh 2025-12-04T09:34:01.9642233Z +++ cd .ci/pytorch 2025-12-04T09:34:01.9642464Z +++ pwd -P 2025-12-04T09:34:01.9643136Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-12-04T09:34:01.9645049Z ++ [[ linux-jammy-py3.10-gcc11-build == *-pch* ]] 2025-12-04T09:34:01.9645318Z ++ which sccache 2025-12-04T09:34:01.9661420Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-12-04T09:34:01.9661737Z ++ sccache --stop-server 2025-12-04T09:34:01.9685915Z ++ true 2025-12-04T09:34:01.9686286Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-12-04T09:34:01.9696935Z ++ trap_add sccache_epilogue EXIT 2025-12-04T09:34:01.9697350Z ++ trap_add_cmd=sccache_epilogue 2025-12-04T09:34:01.9701929Z ++ shift 2025-12-04T09:34:01.9706727Z ++ for trap_add_name in "$@" 2025-12-04T09:34:01.9711507Z ++++ trap -p EXIT 2025-12-04T09:34:01.9713450Z +++ eval 'extract_trap_cmd ' 2025-12-04T09:34:01.9713736Z ++++ extract_trap_cmd 2025-12-04T09:34:01.9713994Z ++++ printf '%s\n' '' 2025-12-04T09:34:01.9714174Z +++ printf '%s\n' sccache_epilogue 2025-12-04T09:34:01.9714430Z ++ trap -- ' 2025-12-04T09:34:01.9717999Z sccache_epilogue' EXIT 2025-12-04T09:34:01.9718220Z ++ [[ -n 1 ]] 2025-12-04T09:34:01.9718590Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-12-04T09:34:01.9718956Z Skipping sccache server initialization, setting environment variables 2025-12-04T09:34:01.9719249Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:34:01.9719436Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:34:01.9719652Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:34:01.9719915Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:34:01.9727400Z ++ export RUST_LOG=sccache::server=error 2025-12-04T09:34:01.9727725Z ++ RUST_LOG=sccache::server=error 2025-12-04T09:34:01.9727940Z ++ sccache --zero-stats 2025-12-04T09:34:02.1327789Z Statistics zeroed. 2025-12-04T09:34:02.1329374Z ++ which ccache 2025-12-04T09:34:02.1363798Z + [[ linux-jammy-py3.10-gcc11-build != *rocm* ]] 2025-12-04T09:34:02.1364101Z + [[ linux-jammy-py3.10-gcc11-build != *s390x* ]] 2025-12-04T09:34:02.1364963Z + [[ -d /var/lib/jenkins/workspace ]] 2025-12-04T09:34:02.1366338Z ++ stat -c %u /var/lib/jenkins/workspace 2025-12-04T09:34:02.1371278Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-12-04T09:34:02.1371522Z + trap_add cleanup_workspace EXIT 2025-12-04T09:34:02.1371894Z + trap_add_cmd=cleanup_workspace 2025-12-04T09:34:02.1372096Z + shift 2025-12-04T09:34:02.1372442Z + for trap_add_name in "$@" 2025-12-04T09:34:02.1384729Z +++ trap -p EXIT 2025-12-04T09:34:02.1384970Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-12-04T09:34:02.1385200Z sccache_epilogue'\'' EXIT' 2025-12-04T09:34:02.1385395Z +++ extract_trap_cmd trap -- ' 2025-12-04T09:34:02.1385590Z sccache_epilogue' EXIT 2025-12-04T09:34:02.1385757Z +++ printf '%s\n' ' 2025-12-04T09:34:02.1385921Z sccache_epilogue' 2025-12-04T09:34:02.1386309Z ++ printf '%s\n' cleanup_workspace 2025-12-04T09:34:02.1386508Z + trap -- ' 2025-12-04T09:34:02.1386663Z sccache_epilogue 2025-12-04T09:34:02.1386833Z cleanup_workspace' EXIT 2025-12-04T09:34:02.1387037Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-12-04T09:34:02.5849169Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-12-04T09:34:02.5869114Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T09:34:02.5869366Z + echo 'Environment variables:' 2025-12-04T09:34:02.5870303Z Environment variables: 2025-12-04T09:34:02.5870580Z + env 2025-12-04T09:34:02.5881972Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:34:02.5883886Z CONTINUE_THROUGH_ERROR=True 2025-12-04T09:34:02.5884162Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-12-04T09:34:02.5885010Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T09:34:02.5885400Z HOSTNAME=b4276b4c571f 2025-12-04T09:34:02.5885809Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.5886630Z GITHUB_ACTION=__run_3 2025-12-04T09:34:02.5887103Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-12-04T09:34:02.5887316Z GITHUB_RUN_NUMBER=67027 2025-12-04T09:34:02.5887529Z TEST_CONFIG=cpu_inductor_huggingface 2025-12-04T09:34:02.5887994Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T09:34:02.5888209Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T09:34:02.5888418Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:34:02.5888841Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T09:34:02.5889057Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-12-04T09:34:02.5889251Z GITHUB_REF_TYPE=branch 2025-12-04T09:34:02.5889454Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:34:02.5889669Z XLA_CUDA= 2025-12-04T09:34:02.5889822Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T09:34:02.5890082Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T09:34:02.5890541Z *** 2025-12-04T09:34:02.5890700Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T09:34:02.5890876Z GITHUB_ACTIONS=true 2025-12-04T09:34:02.5891073Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:34:02.5891320Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:34:02.5891551Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:34:02.5892143Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/heads/main 2025-12-04T09:34:02.5892476Z UCC_HOME=/usr 2025-12-04T09:34:02.5892632Z VERBOSE_TEST_LOGS=False 2025-12-04T09:34:02.5892813Z GITHUB_REF=refs/heads/main 2025-12-04T09:34:02.5892985Z SHARD_NUMBER=1 2025-12-04T09:34:02.5893141Z GITHUB_REF_PROTECTED=true 2025-12-04T09:34:02.5893327Z HOME=/var/lib/jenkins 2025-12-04T09:34:02.5893510Z GITHUB_API_URL=https://api.github.com 2025-12-04T09:34:02.5893720Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T09:34:02.5893901Z UCX_COMMIT= 2025-12-04T09:34:02.5894033Z USE_SYSTEM_NCCL=1 2025-12-04T09:34:02.5894183Z NUM_TEST_SHARDS=1 2025-12-04T09:34:02.5894327Z UCX_HOME=/usr 2025-12-04T09:34:02.5894656Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.5895180Z JOB_NAME=periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:34:02.5895699Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.5896200Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T09:34:02.5896504Z GITHUB_EVENT_NAME=schedule 2025-12-04T09:34:02.5896672Z DASHBOARD_TAG= 2025-12-04T09:34:02.5896824Z GITHUB_RUN_ID=19923066595 2025-12-04T09:34:02.5896985Z INSTALLED_OPENBLAS= 2025-12-04T09:34:02.5897377Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.5897795Z GITHUB_ACTOR=pytorchmergebot 2025-12-04T09:34:02.5897962Z PR_NUMBER= 2025-12-04T09:34:02.5898105Z DESIRED_CUDA= 2025-12-04T09:34:02.5898255Z GITHUB_RUN_ATTEMPT=1 2025-12-04T09:34:02.5898418Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T09:34:02.5898629Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T09:34:02.5898853Z TERM=vt100 2025-12-04T09:34:02.5898988Z INSTALLED_VISION=yes 2025-12-04T09:34:02.5899148Z BRANCH=main 2025-12-04T09:34:02.5899297Z SCCACHE_REGION=us-east-1 2025-12-04T09:34:02.5899467Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T09:34:02.5899650Z BUILD_AOT_INDUCTOR_TEST=yes 2025-12-04T09:34:02.5899927Z CUDA_PATH=/usr/local/cuda 2025-12-04T09:34:02.5900250Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T09:34:02.5900617Z GITHUB_SERVER_URL=https://github.com 2025-12-04T09:34:02.5900805Z UCC_COMMIT= 2025-12-04T09:34:02.5900944Z REENABLED_ISSUES= 2025-12-04T09:34:02.5901083Z DOCS=yes 2025-12-04T09:34:02.5901216Z SHLVL=1 2025-12-04T09:34:02.5901347Z MAX_JOBS=30 2025-12-04T09:34:02.5901482Z GITHUB_ACTOR_ID=97764156 2025-12-04T09:34:02.5901697Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:34:02.5901926Z GITHUB_REF_NAME=main 2025-12-04T09:34:02.5902150Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:34:02.5902398Z GITHUB_JOB=test 2025-12-04T09:34:02.5902549Z NO_TEST_TIMEOUT=False 2025-12-04T09:34:02.5902750Z TD_DISTRIBUTED=False 2025-12-04T09:34:02.5902915Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T09:34:02.5903102Z GITHUB_RETENTION_DAYS=90 2025-12-04T09:34:02.5903271Z OPENSSL_DIR=/opt/openssl 2025-12-04T09:34:02.5903431Z GITHUB_ACTION_REPOSITORY= 2025-12-04T09:34:02.5903871Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:34:02.5904309Z GITHUB_BASE_REF= 2025-12-04T09:34:02.5904450Z INSTALLED_ACL= 2025-12-04T09:34:02.5904699Z ARTIFACTS_FILE_SUFFIX=test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268 2025-12-04T09:34:02.5904982Z CI=true 2025-12-04T09:34:02.5905121Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T09:34:02.5905352Z RUST_LOG=sccache::server=error 2025-12-04T09:34:02.5905524Z JOB_ID=57118563268 2025-12-04T09:34:02.5905662Z GITHUB_HEAD_REF= 2025-12-04T09:34:02.5905805Z GITHUB_ACTION_REF= 2025-12-04T09:34:02.5905986Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T09:34:02.5906193Z TEST_SHOWLOCALS=False 2025-12-04T09:34:02.5906361Z GITHUB_WORKFLOW=inductor-periodic 2025-12-04T09:34:02.5906553Z DEBIAN_FRONTEND=noninteractive 2025-12-04T09:34:02.5906911Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.5907255Z NO_TD=False 2025-12-04T09:34:02.5907406Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T09:34:02.5907598Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T09:34:02.5907780Z _=/usr/bin/env 2025-12-04T09:34:02.5907979Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-12-04T09:34:02.6139189Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-12-04T09:34:02.6139673Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-12-04T09:34:02.6140055Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-12-04T09:34:02.6140499Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-12-04T09:34:02.6140891Z + BUILD_DIR=build 2025-12-04T09:34:02.6141167Z + BUILD_RENAMED_DIR=build_renamed 2025-12-04T09:34:02.6141427Z + BUILD_BIN_DIR=build/bin 2025-12-04T09:34:02.6141621Z + SHARD_NUMBER=1 2025-12-04T09:34:02.6141781Z + NUM_TEST_SHARDS=1 2025-12-04T09:34:02.6141976Z + export TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:34:02.6142230Z + TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:34:02.6142462Z + export VALGRIND=ON 2025-12-04T09:34:02.6142652Z + VALGRIND=ON 2025-12-04T09:34:02.6142899Z + [[ linux-jammy-py3.10-gcc11-build == *clang9* ]] 2025-12-04T09:34:02.6143196Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T09:34:02.6143454Z + detect_cuda_arch 2025-12-04T09:34:02.6143678Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T09:34:02.6143977Z + [[ linux-jammy-py3.10-gcc11-build == *s390x* ]] 2025-12-04T09:34:02.6144206Z + [[ 0 == \1 ]] 2025-12-04T09:34:02.6144371Z + [[ True == \1 ]] 2025-12-04T09:34:02.6144597Z + [[ linux-jammy-py3.10-gcc11-build != *bazel* ]] 2025-12-04T09:34:02.6144850Z ++ realpath build/custom_test_artifacts 2025-12-04T09:34:02.6155304Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-12-04T09:34:02.6155682Z + [[ -n '' ]] 2025-12-04T09:34:02.6155878Z + echo 'Environment variables' 2025-12-04T09:34:02.6156080Z Environment variables 2025-12-04T09:34:02.6156277Z + env 2025-12-04T09:34:02.6180968Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:34:02.6181485Z CONTINUE_THROUGH_ERROR=True 2025-12-04T09:34:02.6181791Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-12-04T09:34:02.6182401Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T09:34:02.6182628Z HOSTNAME=b4276b4c571f 2025-12-04T09:34:02.6183060Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.6183488Z GITHUB_ACTION=__run_3 2025-12-04T09:34:02.6183922Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-12-04T09:34:02.6184144Z GITHUB_RUN_NUMBER=67027 2025-12-04T09:34:02.6184351Z TEST_CONFIG=cpu_inductor_huggingface 2025-12-04T09:34:02.6184592Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T09:34:02.6184835Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T09:34:02.6185066Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:34:02.6185349Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T09:34:02.6185569Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-12-04T09:34:02.6185782Z GITHUB_REF_TYPE=branch 2025-12-04T09:34:02.6185997Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:34:02.6186219Z XLA_CUDA= 2025-12-04T09:34:02.6186384Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T09:34:02.6186691Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T09:34:02.6186924Z *** 2025-12-04T09:34:02.6187132Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T09:34:02.6187329Z GITHUB_ACTIONS=true 2025-12-04T09:34:02.6187543Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:34:02.6187837Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:34:02.6188078Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:34:02.6188432Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/heads/main 2025-12-04T09:34:02.6188754Z UCC_HOME=/usr 2025-12-04T09:34:02.6188910Z TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:34:02.6189102Z VERBOSE_TEST_LOGS=False 2025-12-04T09:34:02.6189284Z GITHUB_REF=refs/heads/main 2025-12-04T09:34:02.6189462Z SHARD_NUMBER=1 2025-12-04T09:34:02.6189633Z GITHUB_REF_PROTECTED=true 2025-12-04T09:34:02.6189808Z HOME=/var/lib/jenkins 2025-12-04T09:34:02.6189992Z GITHUB_API_URL=https://api.github.com 2025-12-04T09:34:02.6190207Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T09:34:02.6190397Z UCX_COMMIT= 2025-12-04T09:34:02.6190536Z USE_SYSTEM_NCCL=1 2025-12-04T09:34:02.6190693Z NUM_TEST_SHARDS=1 2025-12-04T09:34:02.6190847Z UCX_HOME=/usr 2025-12-04T09:34:02.6191207Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.6191745Z JOB_NAME=periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:34:02.6192312Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.6192814Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T09:34:02.6193110Z GITHUB_EVENT_NAME=schedule 2025-12-04T09:34:02.6193286Z DASHBOARD_TAG= 2025-12-04T09:34:02.6193445Z GITHUB_RUN_ID=19923066595 2025-12-04T09:34:02.6193611Z INSTALLED_OPENBLAS= 2025-12-04T09:34:02.6193981Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.6194385Z GITHUB_ACTOR=pytorchmergebot 2025-12-04T09:34:02.6194566Z PR_NUMBER= 2025-12-04T09:34:02.6194702Z DESIRED_CUDA= 2025-12-04T09:34:02.6194855Z GITHUB_RUN_ATTEMPT=1 2025-12-04T09:34:02.6195018Z VALGRIND=ON 2025-12-04T09:34:02.6195177Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T09:34:02.6195406Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T09:34:02.6195668Z TERM=vt100 2025-12-04T09:34:02.6195897Z INSTALLED_VISION=yes 2025-12-04T09:34:02.6196066Z BRANCH=main 2025-12-04T09:34:02.6196218Z SCCACHE_REGION=us-east-1 2025-12-04T09:34:02.6196396Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T09:34:02.6196587Z BUILD_AOT_INDUCTOR_TEST=yes 2025-12-04T09:34:02.6196769Z CUDA_PATH=/usr/local/cuda 2025-12-04T09:34:02.6197091Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T09:34:02.6197445Z GITHUB_SERVER_URL=https://github.com 2025-12-04T09:34:02.6197644Z UCC_COMMIT= 2025-12-04T09:34:02.6197785Z REENABLED_ISSUES= 2025-12-04T09:34:02.6197931Z DOCS=yes 2025-12-04T09:34:02.6198069Z SHLVL=1 2025-12-04T09:34:02.6198204Z MAX_JOBS=30 2025-12-04T09:34:02.6198342Z GITHUB_ACTOR_ID=97764156 2025-12-04T09:34:02.6198568Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:34:02.6198911Z GITHUB_REF_NAME=main 2025-12-04T09:34:02.6199152Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:34:02.6199415Z GITHUB_JOB=test 2025-12-04T09:34:02.6199569Z NO_TEST_TIMEOUT=False 2025-12-04T09:34:02.6199726Z TD_DISTRIBUTED=False 2025-12-04T09:34:02.6199900Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T09:34:02.6200097Z GITHUB_RETENTION_DAYS=90 2025-12-04T09:34:02.6200265Z OPENSSL_DIR=/opt/openssl 2025-12-04T09:34:02.6200443Z GITHUB_ACTION_REPOSITORY= 2025-12-04T09:34:02.6200914Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:34:02.6201379Z GITHUB_BASE_REF= 2025-12-04T09:34:02.6201530Z INSTALLED_ACL= 2025-12-04T09:34:02.6201799Z ARTIFACTS_FILE_SUFFIX=test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268 2025-12-04T09:34:02.6202095Z CI=true 2025-12-04T09:34:02.6202238Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T09:34:02.6202489Z RUST_LOG=sccache::server=error 2025-12-04T09:34:02.6202669Z JOB_ID=57118563268 2025-12-04T09:34:02.6202813Z GITHUB_HEAD_REF= 2025-12-04T09:34:02.6202971Z GITHUB_ACTION_REF= 2025-12-04T09:34:02.6203167Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T09:34:02.6203385Z TEST_SHOWLOCALS=False 2025-12-04T09:34:02.6203563Z GITHUB_WORKFLOW=inductor-periodic 2025-12-04T09:34:02.6203765Z DEBIAN_FRONTEND=noninteractive 2025-12-04T09:34:02.6204133Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_9ba1e37f-ed93-4d4e-8d2a-be715daaa6e0 2025-12-04T09:34:02.6204506Z NO_TD=False 2025-12-04T09:34:02.6204669Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T09:34:02.6204872Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T09:34:02.6205063Z _=/usr/bin/env 2025-12-04T09:34:02.6205232Z + echo 'Testing pytorch' 2025-12-04T09:34:02.6205395Z Testing pytorch 2025-12-04T09:34:02.6205549Z + export LANG=C.UTF-8 2025-12-04T09:34:02.6205703Z + LANG=C.UTF-8 2025-12-04T09:34:02.6205850Z + PR_NUMBER= 2025-12-04T09:34:02.6206013Z + [[ cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-12-04T09:34:02.6206268Z + [[ cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-12-04T09:34:02.6206504Z + [[ cpu_inductor_huggingface == \s\l\o\w ]] 2025-12-04T09:34:02.6206744Z + [[ linux-jammy-py3.10-gcc11-build == *slow-gradcheck* ]] 2025-12-04T09:34:02.6206997Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T09:34:02.6207227Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T09:34:02.6207443Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T09:34:02.6207665Z + [[ cpu_inductor_huggingface == *crossref* ]] 2025-12-04T09:34:02.6207881Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T09:34:02.6208104Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T09:34:02.6208326Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-12-04T09:34:02.6208547Z + pip_install ninja==1.10.2 2025-12-04T09:34:02.6208777Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-12-04T09:34:02.6209056Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-12-04T09:34:02.9596491Z Collecting ninja==1.10.2 2025-12-04T09:34:02.9807282Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-12-04T09:34:02.9924859Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-12-04T09:34:03.7465042Z Installing collected packages: ninja 2025-12-04T09:34:03.7465347Z Attempting uninstall: ninja 2025-12-04T09:34:03.7472751Z Found existing installation: ninja 1.11.1.4 2025-12-04T09:34:03.7491520Z Uninstalling ninja-1.11.1.4: 2025-12-04T09:34:03.7621651Z Successfully uninstalled ninja-1.11.1.4 2025-12-04T09:34:04.0212517Z Successfully installed ninja-1.10.2 2025-12-04T09:34:04.1251884Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:34:04.1253347Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:34:04.1253992Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-12-04T09:34:04.1254272Z + [[ linux-jammy-py3.10-gcc11-build == *asan* ]] 2025-12-04T09:34:04.1254539Z + [[ linux-jammy-py3.10-gcc11-build == *-debug* ]] 2025-12-04T09:34:04.1254805Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-12-04T09:34:04.1255161Z + echo 'We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass' 2025-12-04T09:34:04.1255596Z We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass 2025-12-04T09:34:04.1255903Z + cd test 2025-12-04T09:34:04.1258558Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-12-04T09:34:04.4052355Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:34:04.4053399Z import pynvml # type: ignore[import] 2025-12-04T09:34:05.2824903Z + [[ cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-12-04T09:34:05.2825431Z + [[ cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-12-04T09:34:05.2830923Z + [[ cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-12-04T09:34:05.2835787Z + DYNAMO_BENCHMARK_FLAGS=() 2025-12-04T09:34:05.2840697Z + [[ cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-12-04T09:34:05.2841016Z + [[ cpu_inductor_huggingface == *dynamo_eager* ]] 2025-12-04T09:34:05.2841248Z + [[ cpu_inductor_huggingface == *aot_eager* ]] 2025-12-04T09:34:05.2841480Z + [[ cpu_inductor_huggingface == *aot_inductor* ]] 2025-12-04T09:34:05.2841766Z + [[ cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-12-04T09:34:05.2842002Z + [[ cpu_inductor_huggingface == *inductor* ]] 2025-12-04T09:34:05.2842238Z + [[ cpu_inductor_huggingface != *perf* ]] 2025-12-04T09:34:05.2842475Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-12-04T09:34:05.2842686Z + [[ cpu_inductor_huggingface == *dynamic* ]] 2025-12-04T09:34:05.2842888Z + [[ cpu_inductor_huggingface == *cpu* ]] 2025-12-04T09:34:05.2843090Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-12-04T09:34:05.3087790Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-12-04T09:34:05.3088131Z + [[ linux-jammy-py3.10-gcc11-build == *-bazel-* ]] 2025-12-04T09:34:05.3088373Z + cd test 2025-12-04T09:34:05.3088609Z + python -c 'import torch; print(torch.__config__.show())' 2025-12-04T09:34:05.5790636Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:34:06.2209653Z import pynvml # type: ignore[import] 2025-12-04T09:34:06.2210329Z PyTorch built with: 2025-12-04T09:34:06.2210538Z - GCC 11.4 2025-12-04T09:34:06.2210692Z - C++ Version: 201703 2025-12-04T09:34:06.2211025Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T09:34:06.2211449Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T09:34:06.2211890Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T09:34:06.2212130Z - LAPACK is enabled (usually provided by MKL) 2025-12-04T09:34:06.2212344Z - NNPACK is enabled 2025-12-04T09:34:06.2212532Z - CPU capability usage: AVX512 2025-12-04T09:34:06.2215234Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-12-04T09:34:06.2218225Z 2025-12-04T09:34:06.4410467Z + cd test 2025-12-04T09:34:06.4410844Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-12-04T09:34:06.7093579Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:34:06.7095120Z import pynvml # type: ignore[import] 2025-12-04T09:34:07.3514562Z ATen/Parallel: 2025-12-04T09:34:07.3514871Z at::get_num_threads() : 16 2025-12-04T09:34:07.3515102Z at::get_num_interop_threads() : 16 2025-12-04T09:34:07.3515329Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T09:34:07.3515537Z omp_get_max_threads() : 16 2025-12-04T09:34:07.3515952Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T09:34:07.3516362Z mkl_get_max_threads() : 16 2025-12-04T09:34:07.3516677Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T09:34:07.3516975Z std::thread::hardware_concurrency() : 32 2025-12-04T09:34:07.3517208Z Environment variables: 2025-12-04T09:34:07.3517399Z OMP_NUM_THREADS : [not set] 2025-12-04T09:34:07.3517584Z MKL_NUM_THREADS : [not set] 2025-12-04T09:34:07.3517779Z ATen parallel backend: OpenMP 2025-12-04T09:34:07.3517913Z 2025-12-04T09:34:07.5697431Z + [[ cpu_inductor_huggingface == *numpy_2* ]] 2025-12-04T09:34:07.5702156Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-12-04T09:34:07.5706223Z + [[ cpu_inductor_huggingface == *backward* ]] 2025-12-04T09:34:07.5712337Z + [[ cpu_inductor_huggingface == *libtorch_agnostic_targetting* ]] 2025-12-04T09:34:07.5714601Z + [[ cpu_inductor_huggingface == *xla* ]] 2025-12-04T09:34:07.5714870Z + [[ cpu_inductor_huggingface == *vllm* ]] 2025-12-04T09:34:07.5715170Z + [[ cpu_inductor_huggingface == *executorch* ]] 2025-12-04T09:34:07.5715434Z + [[ cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-12-04T09:34:07.5715712Z + [[ cpu_inductor_huggingface == \q\u\a\n\t\i\z\a\t\i\o\n ]] 2025-12-04T09:34:07.5716077Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-12-04T09:34:07.5716861Z + [[ cpu_inductor_huggingface == distributed ]] 2025-12-04T09:34:07.5717169Z + [[ cpu_inductor_huggingface == *operator_benchmark* ]] 2025-12-04T09:34:07.5717511Z + [[ cpu_inductor_huggingface == *operator_microbenchmark* ]] 2025-12-04T09:34:07.5717893Z + [[ cpu_inductor_huggingface == *attention_microbenchmark* ]] 2025-12-04T09:34:07.5718246Z + [[ cpu_inductor_huggingface == *inductor_distributed* ]] 2025-12-04T09:34:07.5718548Z + [[ cpu_inductor_huggingface == *inductor-halide* ]] 2025-12-04T09:34:07.5718866Z + [[ cpu_inductor_huggingface == *inductor-pallas* ]] 2025-12-04T09:34:07.5719206Z + [[ cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-12-04T09:34:07.5719516Z + [[ cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-12-04T09:34:07.5719875Z + [[ cpu_inductor_huggingface == *aoti_cross_compile_for_windows* ]] 2025-12-04T09:34:07.5720319Z + [[ cpu_inductor_huggingface == *huggingface* ]] 2025-12-04T09:34:07.5720594Z + install_torchvision 2025-12-04T09:34:07.5720809Z + local orig_preload 2025-12-04T09:34:07.5721043Z + local commit 2025-12-04T09:34:07.5721278Z ++ get_pinned_commit vision 2025-12-04T09:34:07.5721601Z ++ cat .github/ci_commit_pins/vision.txt 2025-12-04T09:34:07.5721905Z + commit=617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:34:07.5722184Z + orig_preload= 2025-12-04T09:34:07.5722385Z + '[' -n '' ']' 2025-12-04T09:34:07.5722644Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T09:34:07.5723170Z + pip_build_and_install git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e dist/vision 2025-12-04T09:34:07.5723757Z + local build_target=git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T09:34:07.5724115Z + local wheel_dir=dist/vision 2025-12-04T09:34:07.5724367Z + local found_whl=0 2025-12-04T09:34:07.5740873Z + for file in "${wheel_dir}"/*.whl 2025-12-04T09:34:07.5741227Z + [[ -f dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl ]] 2025-12-04T09:34:07.5741551Z + found_whl=1 2025-12-04T09:34:07.5741702Z + break 2025-12-04T09:34:07.5741847Z + '[' 1 == 0 ']' 2025-12-04T09:34:07.5742017Z + for file in "${wheel_dir}"/*.whl 2025-12-04T09:34:07.5742328Z + pip_install_whl dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:34:07.5742737Z + args=('dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl') 2025-12-04T09:34:07.5743021Z + local args 2025-12-04T09:34:07.5743279Z + [[ dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl == *\ * ]] 2025-12-04T09:34:07.5743578Z + for path in "${args[@]}" 2025-12-04T09:34:07.5743876Z + echo 'Installing dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl' 2025-12-04T09:34:07.5744302Z Installing dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:34:07.5744755Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:34:07.8415809Z Processing ./dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T09:34:07.8492523Z Installing collected packages: torchvision 2025-12-04T09:34:08.4050632Z Successfully installed torchvision-0.25.0a0+617079d 2025-12-04T09:34:08.4459938Z + '[' -n '' ']' 2025-12-04T09:34:08.4461594Z + id=0 2025-12-04T09:34:08.4467332Z + test_dynamo_benchmark huggingface 0 2025-12-04T09:34:08.4471607Z ++ pwd 2025-12-04T09:34:08.4475960Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-12-04T09:34:08.4480222Z + local suite=huggingface 2025-12-04T09:34:08.4485504Z + shift 2025-12-04T09:34:08.4490046Z + local shard_id=0 2025-12-04T09:34:08.4490267Z + shift 2025-12-04T09:34:08.4490427Z + extra_args=() 2025-12-04T09:34:08.4490618Z + local extra_args 2025-12-04T09:34:08.4490823Z + [[ linux-jammy-py3.10-gcc11-build == *cuda13* ]] 2025-12-04T09:34:08.4491145Z + [[ cpu_inductor_huggingface == *perf_compare* ]] 2025-12-04T09:34:08.4491399Z + [[ cpu_inductor_huggingface == *perf* ]] 2025-12-04T09:34:08.4492258Z + [[ cpu_inductor_huggingface == *cpu* ]] 2025-12-04T09:34:08.4492502Z + local dt=float32 2025-12-04T09:34:08.4492689Z + [[ cpu_inductor_huggingface == *amp* ]] 2025-12-04T09:34:08.4492931Z + [[ cpu_inductor_huggingface == *freezing* ]] 2025-12-04T09:34:08.4493263Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-12-04T09:34:08.4493554Z ++ pwd 2025-12-04T09:34:08.4493777Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-12-04T09:34:08.4494078Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-12-04T09:34:08.4494338Z + local name=inference 2025-12-04T09:34:08.4494517Z + shift 2025-12-04T09:34:08.4494659Z + local suite=huggingface 2025-12-04T09:34:08.4494830Z + shift 2025-12-04T09:34:08.4494970Z + local shard_id=0 2025-12-04T09:34:08.4495224Z + shift 2025-12-04T09:34:08.4495374Z + partition_flags=() 2025-12-04T09:34:08.4495552Z + local partition_flags 2025-12-04T09:34:08.4495731Z + [[ -n 1 ]] 2025-12-04T09:34:08.4495881Z + [[ -n 0 ]] 2025-12-04T09:34:08.4496151Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-12-04T09:34:08.4496483Z + [[ cpu_inductor_huggingface == *perf_compare* ]] 2025-12-04T09:34:08.4496716Z + [[ cpu_inductor_huggingface == *perf* ]] 2025-12-04T09:34:08.4496940Z + [[ cpu_inductor_huggingface == *_avx2* ]] 2025-12-04T09:34:08.4497162Z + [[ cpu_inductor_huggingface == *_avx512* ]] 2025-12-04T09:34:08.4497906Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-12-04T09:34:09.1564825Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:34:09.1566518Z import pynvml # type: ignore[import] 2025-12-04T09:34:12.3193196Z 2025-12-04T09:34:12.3193884Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-12-04T09:35:42.3953260Z 2025-12-04T09:35:42.3953361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.3953857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.3954320Z layer_outputs = layer_module( 2025-12-04T09:35:42.3954662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.3955026Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.3955431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.3955831Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.3956229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.3956627Z self_outputs = self.self( 2025-12-04T09:35:42.3957012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.3957433Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.3957918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.3958491Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.3958734Z 2025-12-04T09:35:42.3958833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.3959342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.3959827Z layer_outputs = layer_module( 2025-12-04T09:35:42.3960152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.3960490Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.3960886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.3961290Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.3961727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.3962110Z self_outputs = self.self( 2025-12-04T09:35:42.3962498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.3962926Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.3963406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.3963976Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.3964209Z 2025-12-04T09:35:42.3964308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.3964805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.3966088Z layer_outputs = layer_module( 2025-12-04T09:35:42.3966420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.3966765Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.3967161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.3967548Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.3967943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.3968328Z self_outputs = self.self( 2025-12-04T09:35:42.3968709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.3969135Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.3969625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.3970192Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.3970424Z 2025-12-04T09:35:42.3970509Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.3970713Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.3970915Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.3971112Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.3971334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.3971911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.3972633Z layer_outputs = layer_module( 2025-12-04T09:35:42.3973019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.3973428Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.3973886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.3974305Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.3974751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.3975201Z self_outputs = self.self( 2025-12-04T09:35:42.3975642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.3976133Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.3976683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.3977379Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.3977960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.3978485Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.3978647Z 2025-12-04T09:35:42.3978731Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.3978984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.3979543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.3980060Z layer_outputs = layer_module( 2025-12-04T09:35:42.3980413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.3980827Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.3981254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.3981669Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.3982087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.3982503Z self_outputs = self.self( 2025-12-04T09:35:42.3982908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.3983319Z attn_scores += diagonal_mask 2025-12-04T09:35:42.3983453Z 2025-12-04T09:35:42.3983559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.3984077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.3984573Z layer_outputs = layer_module( 2025-12-04T09:35:42.3984921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.3985289Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.3985710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.3986123Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.3986543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.3986950Z self_outputs = self.self( 2025-12-04T09:35:42.3987349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.3987765Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.3987906Z 2025-12-04T09:35:42.3988015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.3988530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.3989016Z layer_outputs = layer_module( 2025-12-04T09:35:42.3989362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.3989727Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.3990145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.3990555Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.3990972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.3991386Z self_outputs = self.self( 2025-12-04T09:35:42.3991825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.3992274Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.3992778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.3993346Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.3993768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.3994109Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.3994262Z 2025-12-04T09:35:42.3994361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.3994902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.3995377Z layer_outputs = layer_module( 2025-12-04T09:35:42.3995713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.3996075Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.3996482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.3996885Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.3997298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.3997699Z self_outputs = self.self( 2025-12-04T09:35:42.3998085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.3998529Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.3999046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.3999579Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4000079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4000529Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4000863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4001205Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4001354Z 2025-12-04T09:35:42.4001453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4001955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4002432Z layer_outputs = layer_module( 2025-12-04T09:35:42.4002777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4003127Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4003538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4003943Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4004348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4004741Z self_outputs = self.self( 2025-12-04T09:35:42.4005132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4005622Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4006140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4006683Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4006891Z 2025-12-04T09:35:42.4006993Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4007495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4007966Z layer_outputs = layer_module( 2025-12-04T09:35:42.4008300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4008696Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4009111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4009513Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4009927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4010334Z self_outputs = self.self( 2025-12-04T09:35:42.4010732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4011186Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4011836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4012450Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4012678Z 2025-12-04T09:35:42.4012790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4013285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4013759Z layer_outputs = layer_module( 2025-12-04T09:35:42.4014095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4014452Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4014855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4015262Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4015666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4016059Z self_outputs = self.self( 2025-12-04T09:35:42.4016452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4016967Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4017201Z 2025-12-04T09:35:42.4017310Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4017800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4018275Z layer_outputs = layer_module( 2025-12-04T09:35:42.4018619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4018975Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4019412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4019819Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4020225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4020661Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4021087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4021500Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4021635Z 2025-12-04T09:35:42.4021741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4022228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4022739Z layer_outputs = layer_module( 2025-12-04T09:35:42.4023070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4023403Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4023779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4024177Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4024553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4024918Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4025298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4025725Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4026140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4026523Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4026656Z 2025-12-04T09:35:42.4026751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4027221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4027661Z layer_outputs = layer_module( 2025-12-04T09:35:42.4027968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4028303Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4028684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4029074Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4029441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4029805Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4030185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4030604Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4031006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4031423Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4031779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4032095Z return self.act(input) 2025-12-04T09:35:42.4032204Z 2025-12-04T09:35:42.4032298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4032808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4033259Z layer_outputs = layer_module( 2025-12-04T09:35:42.4033577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4033910Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4034298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4034688Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4035052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4035455Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4035842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4036269Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4036693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4037089Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4037216Z 2025-12-04T09:35:42.4037317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4037779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4038226Z layer_outputs = layer_module( 2025-12-04T09:35:42.4038545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4038885Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4039264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4039650Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4040038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4040418Z self_outputs = self.self( 2025-12-04T09:35:42.4040779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4041166Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4041290Z 2025-12-04T09:35:42.4041392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4041854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4042308Z layer_outputs = layer_module( 2025-12-04T09:35:42.4042627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4042960Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4043335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4043719Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4044100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4044475Z self_outputs = self.self( 2025-12-04T09:35:42.4044831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4045245Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4045738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4046282Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4046516Z 2025-12-04T09:35:42.4046610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4047084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4047529Z layer_outputs = layer_module( 2025-12-04T09:35:42.4047842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4048179Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4048600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4048994Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4049377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4049765Z self_outputs = self.self( 2025-12-04T09:35:42.4050143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4050538Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4050665Z 2025-12-04T09:35:42.4050762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4051300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4051912Z layer_outputs = layer_module( 2025-12-04T09:35:42.4052291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4052684Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4053139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4053591Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4054025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4054433Z self_outputs = self.self( 2025-12-04T09:35:42.4054864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4055342Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4055880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4056523Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4056787Z 2025-12-04T09:35:42.4056903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4057453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4057968Z layer_outputs = layer_module( 2025-12-04T09:35:42.4058338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4058728Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4059161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4059614Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4060103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4060609Z self_outputs = self.self( 2025-12-04T09:35:42.4061029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4061506Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4062031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4062663Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4062908Z 2025-12-04T09:35:42.4063007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4063530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4063993Z layer_outputs = layer_module( 2025-12-04T09:35:42.4064322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4064660Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4065060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4065456Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4065839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4066230Z self_outputs = self.self( 2025-12-04T09:35:42.4066608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4067026Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4067603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4068164Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4068395Z 2025-12-04T09:35:42.4068473Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4068680Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4068873Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4069071Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4069294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4069771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4070235Z layer_outputs = layer_module( 2025-12-04T09:35:42.4070567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4070911Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4071300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4071695Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4072083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4072635Z self_outputs = self.self( 2025-12-04T09:35:42.4073017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4073455Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4074004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4074525Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4075027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4075499Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4075650Z 2025-12-04T09:35:42.4075733Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4075965Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4076454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4076917Z layer_outputs = layer_module( 2025-12-04T09:35:42.4077318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4077655Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4078049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4078447Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4078838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4079221Z self_outputs = self.self( 2025-12-04T09:35:42.4079599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4079989Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4080102Z 2025-12-04T09:35:42.4080197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4080702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4081160Z layer_outputs = layer_module( 2025-12-04T09:35:42.4081485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4081819Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4082213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4082602Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4082987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4083364Z self_outputs = self.self( 2025-12-04T09:35:42.4083736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4084132Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4084259Z 2025-12-04T09:35:42.4084355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4084838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4085293Z layer_outputs = layer_module( 2025-12-04T09:35:42.4085615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4085946Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4086350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4086753Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4087154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4087600Z self_outputs = self.self( 2025-12-04T09:35:42.4087997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4088418Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4088555Z 2025-12-04T09:35:42.4088657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4089162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4089642Z layer_outputs = layer_module( 2025-12-04T09:35:42.4089988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4090340Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4090826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4091249Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4091733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4092152Z self_outputs = self.self( 2025-12-04T09:35:42.4092553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4093019Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4093536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4094106Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4094533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4094883Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4095031Z 2025-12-04T09:35:42.4095130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4095633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4096101Z layer_outputs = layer_module( 2025-12-04T09:35:42.4096438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4096786Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4097191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4097599Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4098007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4098397Z self_outputs = self.self( 2025-12-04T09:35:42.4098783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4099232Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4099742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4100267Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4100763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4101221Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4101582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4101934Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4102084Z 2025-12-04T09:35:42.4102183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4102678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4103146Z layer_outputs = layer_module( 2025-12-04T09:35:42.4103482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4103833Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4104238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4104680Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4105086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4105484Z self_outputs = self.self( 2025-12-04T09:35:42.4105868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4106300Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4106798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4107330Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4107520Z 2025-12-04T09:35:42.4107625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4108105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4108566Z layer_outputs = layer_module( 2025-12-04T09:35:42.4108906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4109247Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4109641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4110036Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4110428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4110809Z self_outputs = self.self( 2025-12-04T09:35:42.4111183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4111619Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4112114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4112632Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4112829Z 2025-12-04T09:35:42.4112933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4113416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4113879Z layer_outputs = layer_module( 2025-12-04T09:35:42.4114195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4114540Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4114967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4115359Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4115762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4116150Z self_outputs = self.self( 2025-12-04T09:35:42.4116536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4117027Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4117263Z 2025-12-04T09:35:42.4117360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4117851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4118343Z layer_outputs = layer_module( 2025-12-04T09:35:42.4118663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4119012Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4119399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4119794Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4120180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4120607Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4121030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4121430Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4121568Z 2025-12-04T09:35:42.4121668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4122155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4122610Z layer_outputs = layer_module( 2025-12-04T09:35:42.4122927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4123274Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4123664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4124070Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4124446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4124838Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4125246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4125681Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4126120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4126535Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4126665Z 2025-12-04T09:35:42.4126771Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4127261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4127740Z layer_outputs = layer_module( 2025-12-04T09:35:42.4128079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4128485Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4128887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4129298Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4129690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4130075Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4130467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4130903Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4131335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4131914Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4132345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4132724Z return self.act(input) 2025-12-04T09:35:42.4132840Z 2025-12-04T09:35:42.4132952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4133463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4133953Z layer_outputs = layer_module( 2025-12-04T09:35:42.4134298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4134667Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4135062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4135470Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4135859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4136242Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4136637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4137091Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4137521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4137908Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4138043Z 2025-12-04T09:35:42.4138138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4138618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4139071Z layer_outputs = layer_module( 2025-12-04T09:35:42.4139386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4139723Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4140108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4140495Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4140872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4141255Z self_outputs = self.self( 2025-12-04T09:35:42.4141629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4142016Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4142203Z 2025-12-04T09:35:42.4142300Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4142776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4143229Z layer_outputs = layer_module( 2025-12-04T09:35:42.4143545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4143882Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4144291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4144678Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4145087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4145469Z self_outputs = self.self( 2025-12-04T09:35:42.4145838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4146244Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4146709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4147252Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4147473Z 2025-12-04T09:35:42.4147576Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4148041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4148493Z layer_outputs = layer_module( 2025-12-04T09:35:42.4148814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4149153Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4149531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4149917Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4150302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4150681Z self_outputs = self.self( 2025-12-04T09:35:42.4151043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4151426Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4151552Z 2025-12-04T09:35:42.4151652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4152118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4152565Z layer_outputs = layer_module( 2025-12-04T09:35:42.4152884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4153217Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4153593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4153980Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4154363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4154741Z self_outputs = self.self( 2025-12-04T09:35:42.4155127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4155541Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4156004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4156546Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4156767Z 2025-12-04T09:35:42.4156863Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4157336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4157780Z layer_outputs = layer_module( 2025-12-04T09:35:42.4158128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4158464Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4158846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4159228Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4159599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4159975Z self_outputs = self.self( 2025-12-04T09:35:42.4160340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4160744Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4161194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4161735Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4161954Z 2025-12-04T09:35:42.4162056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4162526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4162962Z layer_outputs = layer_module( 2025-12-04T09:35:42.4163281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4163622Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4163998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4164383Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4164768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4165146Z self_outputs = self.self( 2025-12-04T09:35:42.4165503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4165908Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4166368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4166900Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4167120Z 2025-12-04T09:35:42.4167195Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4167393Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4167582Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4167765Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4167985Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4168523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4168987Z layer_outputs = layer_module( 2025-12-04T09:35:42.4169315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4169666Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4170064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4170462Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4170847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4171281Z self_outputs = self.self( 2025-12-04T09:35:42.4171738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4172226Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4172849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4173398Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4173932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4174407Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4174566Z 2025-12-04T09:35:42.4174645Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4174889Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4175387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4175846Z layer_outputs = layer_module( 2025-12-04T09:35:42.4176176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4176521Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4176921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4177312Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4177702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4178091Z self_outputs = self.self( 2025-12-04T09:35:42.4178466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4178860Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4178981Z 2025-12-04T09:35:42.4179075Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4179561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4180012Z layer_outputs = layer_module( 2025-12-04T09:35:42.4180340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4180689Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4181084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4181476Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4181940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4182336Z self_outputs = self.self( 2025-12-04T09:35:42.4182711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4183117Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4183252Z 2025-12-04T09:35:42.4183350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4183851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4184315Z layer_outputs = layer_module( 2025-12-04T09:35:42.4184657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4185079Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4185489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4185890Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4186293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4186698Z self_outputs = self.self( 2025-12-04T09:35:42.4187075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4187489Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4187634Z 2025-12-04T09:35:42.4187734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4188234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4188703Z layer_outputs = layer_module( 2025-12-04T09:35:42.4189047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4189401Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4189802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4190196Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4190595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4190991Z self_outputs = self.self( 2025-12-04T09:35:42.4191370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4191802Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4192312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4192876Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4193278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4193619Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4193774Z 2025-12-04T09:35:42.4193873Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4194363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4194820Z layer_outputs = layer_module( 2025-12-04T09:35:42.4195154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4195512Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4195951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4196336Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4196731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4197109Z self_outputs = self.self( 2025-12-04T09:35:42.4197472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4197887Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4198373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4198912Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4199380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4199817Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4200136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4200461Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4200603Z 2025-12-04T09:35:42.4200698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4201174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4201634Z layer_outputs = layer_module( 2025-12-04T09:35:42.4201970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4202310Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4202708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4203110Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4203506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4203898Z self_outputs = self.self( 2025-12-04T09:35:42.4204265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4204686Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4205166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4205698Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4205894Z 2025-12-04T09:35:42.4205988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4206531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4207196Z layer_outputs = layer_module( 2025-12-04T09:35:42.4207610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4208017Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4208529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4208965Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4228889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4229649Z self_outputs = self.self( 2025-12-04T09:35:42.4230072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4230515Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4231037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4231569Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4231763Z 2025-12-04T09:35:42.4231879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4232367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4232906Z layer_outputs = layer_module( 2025-12-04T09:35:42.4233255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4233615Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4234022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4234453Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4234852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4235231Z self_outputs = self.self( 2025-12-04T09:35:42.4235613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4236130Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4236359Z 2025-12-04T09:35:42.4236473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4236963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4237436Z layer_outputs = layer_module( 2025-12-04T09:35:42.4237776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4238131Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4238534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4238924Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4239311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4239734Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4240149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4240556Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4240689Z 2025-12-04T09:35:42.4240799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4241281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4241747Z layer_outputs = layer_module( 2025-12-04T09:35:42.4242081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4242429Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4242827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4243261Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4243642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4244017Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4244401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4244829Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4245247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4245641Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4245769Z 2025-12-04T09:35:42.4245898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4246378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4246828Z layer_outputs = layer_module( 2025-12-04T09:35:42.4247140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4247482Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4247868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4248261Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4248623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4248991Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4249381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4249799Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4250204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4250622Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4250981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4251299Z return self.act(input) 2025-12-04T09:35:42.4251419Z 2025-12-04T09:35:42.4251617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4252149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4252644Z layer_outputs = layer_module( 2025-12-04T09:35:42.4252981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4253326Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4253717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4254111Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4254476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4254845Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4255233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4255663Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4256083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4256509Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4256639Z 2025-12-04T09:35:42.4256744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4257219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4257678Z layer_outputs = layer_module( 2025-12-04T09:35:42.4258007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4258351Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4258737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4259131Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4259556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4259946Z self_outputs = self.self( 2025-12-04T09:35:42.4260312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4260707Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4260832Z 2025-12-04T09:35:42.4260940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4261411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4261866Z layer_outputs = layer_module( 2025-12-04T09:35:42.4262188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4262524Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4262912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4263318Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4263714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4264100Z self_outputs = self.self( 2025-12-04T09:35:42.4264486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4264901Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4265372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4265928Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4266173Z 2025-12-04T09:35:42.4266269Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4266763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4267227Z layer_outputs = layer_module( 2025-12-04T09:35:42.4267552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4267896Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4268295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4268684Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4269081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4269475Z self_outputs = self.self( 2025-12-04T09:35:42.4269890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4270282Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4270415Z 2025-12-04T09:35:42.4270513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4270999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4271459Z layer_outputs = layer_module( 2025-12-04T09:35:42.4271777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4272126Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4272708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4273214Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4273603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4273998Z self_outputs = self.self( 2025-12-04T09:35:42.4274373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4274790Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4275269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4275823Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4276053Z 2025-12-04T09:35:42.4276158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4276644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4277103Z layer_outputs = layer_module( 2025-12-04T09:35:42.4277430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4277772Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4278157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4278548Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4278939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4279325Z self_outputs = self.self( 2025-12-04T09:35:42.4279697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4280111Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4280580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4281105Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4281333Z 2025-12-04T09:35:42.4281428Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4281898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4282343Z layer_outputs = layer_module( 2025-12-04T09:35:42.4282653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4282993Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4283434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4283831Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4284208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4284592Z self_outputs = self.self( 2025-12-04T09:35:42.4284963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4285373Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4285845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4286431Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4286656Z 2025-12-04T09:35:42.4286748Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4286952Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4287154Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4287353Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4287574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4288064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4288531Z layer_outputs = layer_module( 2025-12-04T09:35:42.4288862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4289204Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4289609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4290013Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4290411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4290797Z self_outputs = self.self( 2025-12-04T09:35:42.4291176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4291675Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4292185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4292722Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4293257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4293720Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4293864Z 2025-12-04T09:35:42.4293949Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4294169Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4294663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4295130Z layer_outputs = layer_module( 2025-12-04T09:35:42.4295455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4295804Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4296202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4296601Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4296903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4296972Z self_outputs = self.self( 2025-12-04T09:35:42.4297249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4297318Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4297322Z 2025-12-04T09:35:42.4297426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4297763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4297829Z layer_outputs = layer_module( 2025-12-04T09:35:42.4298051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4298158Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4298434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4298505Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4298765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4298838Z self_outputs = self.self( 2025-12-04T09:35:42.4299099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4299174Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4299185Z 2025-12-04T09:35:42.4299282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4299614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4299691Z layer_outputs = layer_module( 2025-12-04T09:35:42.4299902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4299976Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4300247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4300316Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4300584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4300650Z self_outputs = self.self( 2025-12-04T09:35:42.4300911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4301002Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4301005Z 2025-12-04T09:35:42.4301099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4301440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4301507Z layer_outputs = layer_module( 2025-12-04T09:35:42.4301715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4301795Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4302057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4302128Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4302395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4302462Z self_outputs = self.self( 2025-12-04T09:35:42.4302762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4302878Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4303212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4303385Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4303572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4303673Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4303677Z 2025-12-04T09:35:42.4303773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4304143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4304216Z layer_outputs = layer_module( 2025-12-04T09:35:42.4304424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4304503Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4304767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4304837Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4305106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4305171Z self_outputs = self.self( 2025-12-04T09:35:42.4305431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4305547Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4305884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4306017Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4306317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4306403Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4306595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4306690Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4306693Z 2025-12-04T09:35:42.4306796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4307134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4307201Z layer_outputs = layer_module( 2025-12-04T09:35:42.4307416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4307488Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4307759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4307830Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4308093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4308166Z self_outputs = self.self( 2025-12-04T09:35:42.4308427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4308568Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4308910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4309054Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4309058Z 2025-12-04T09:35:42.4309163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4309496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4309564Z layer_outputs = layer_module( 2025-12-04T09:35:42.4309781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4309899Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4310178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4310251Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4310515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4310590Z self_outputs = self.self( 2025-12-04T09:35:42.4310853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4310967Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4311301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4311446Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4311449Z 2025-12-04T09:35:42.4311554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4311887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4311962Z layer_outputs = layer_module( 2025-12-04T09:35:42.4312170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4312253Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4312520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4312601Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4312866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4312932Z self_outputs = self.self( 2025-12-04T09:35:42.4313192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4313372Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4313375Z 2025-12-04T09:35:42.4313471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4313804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4313870Z layer_outputs = layer_module( 2025-12-04T09:35:42.4314073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4314152Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4314415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4314523Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4314786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4314890Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4315161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4315239Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4315242Z 2025-12-04T09:35:42.4315347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4315683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4315778Z layer_outputs = layer_module( 2025-12-04T09:35:42.4315994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4316067Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4316338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4316416Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4316661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4316742Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4317015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4317119Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4317405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4317480Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4317483Z 2025-12-04T09:35:42.4317584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4317913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4317978Z layer_outputs = layer_module( 2025-12-04T09:35:42.4318188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4318260Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4318527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4318607Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4318852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4318930Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4319192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4319289Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4319557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4319660Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4319868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4319935Z return self.act(input) 2025-12-04T09:35:42.4319941Z 2025-12-04T09:35:42.4320036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4320405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4320473Z layer_outputs = layer_module( 2025-12-04T09:35:42.4320688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4320759Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4321023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4321108Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4321353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4321464Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4321727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4321839Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4322102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4322175Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4322178Z 2025-12-04T09:35:42.4322270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4322600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4322667Z layer_outputs = layer_module( 2025-12-04T09:35:42.4322877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4322951Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4323211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4323289Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4323545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4323616Z self_outputs = self.self( 2025-12-04T09:35:42.4323869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4323943Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4323946Z 2025-12-04T09:35:42.4324046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4324370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4324444Z layer_outputs = layer_module( 2025-12-04T09:35:42.4324648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4324718Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4324983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4325051Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4325313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4325383Z self_outputs = self.self( 2025-12-04T09:35:42.4325636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4325739Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4326081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4326252Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4326255Z 2025-12-04T09:35:42.4326356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4326678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4326750Z layer_outputs = layer_module( 2025-12-04T09:35:42.4326952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4327023Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4327319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4327391Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4327655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4327720Z self_outputs = self.self( 2025-12-04T09:35:42.4327976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4328056Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4328060Z 2025-12-04T09:35:42.4328154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4328487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4328563Z layer_outputs = layer_module( 2025-12-04T09:35:42.4328770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4328853Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4329117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4329187Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4329456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4329521Z self_outputs = self.self( 2025-12-04T09:35:42.4329790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4329885Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4330206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4330390Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4330394Z 2025-12-04T09:35:42.4330491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4330830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4330896Z layer_outputs = layer_module( 2025-12-04T09:35:42.4331103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4331182Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4331446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4331601Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4331931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4332003Z self_outputs = self.self( 2025-12-04T09:35:42.4332304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4332412Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4332771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4332958Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4332962Z 2025-12-04T09:35:42.4333063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4333425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4333535Z layer_outputs = layer_module( 2025-12-04T09:35:42.4333763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4333849Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4334142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4334222Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4334491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4334557Z self_outputs = self.self( 2025-12-04T09:35:42.4334836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4334932Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4335258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4335426Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4335430Z 2025-12-04T09:35:42.4335506Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4335588Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4335660Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4335730Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4335832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4336164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4336239Z layer_outputs = layer_module( 2025-12-04T09:35:42.4336449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4336524Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4336797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4336867Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4337130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4337202Z self_outputs = self.self( 2025-12-04T09:35:42.4337461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4337569Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4337891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4338061Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4338386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4338479Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4338483Z 2025-12-04T09:35:42.4338565Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4338662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4339001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4339080Z layer_outputs = layer_module( 2025-12-04T09:35:42.4339327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4339411Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4339686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4339759Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4340045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4340109Z self_outputs = self.self( 2025-12-04T09:35:42.4340371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4340447Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4340450Z 2025-12-04T09:35:42.4340545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4340886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4340955Z layer_outputs = layer_module( 2025-12-04T09:35:42.4341161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4341243Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4341504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4341580Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4341841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4341904Z self_outputs = self.self( 2025-12-04T09:35:42.4342170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4342247Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4342251Z 2025-12-04T09:35:42.4342354Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4342686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4342751Z layer_outputs = layer_module( 2025-12-04T09:35:42.4342963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4343035Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4343298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4343374Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4343638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4343712Z self_outputs = self.self( 2025-12-04T09:35:42.4344006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4344087Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4344090Z 2025-12-04T09:35:42.4344193Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4344523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4344594Z layer_outputs = layer_module( 2025-12-04T09:35:42.4344803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4344874Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4345179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4345251Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4345520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4345585Z self_outputs = self.self( 2025-12-04T09:35:42.4345844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4345962Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4346294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4346456Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4346649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4346743Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4346747Z 2025-12-04T09:35:42.4346850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4347180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4347246Z layer_outputs = layer_module( 2025-12-04T09:35:42.4347461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4347533Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4347801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4347868Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4348148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4348216Z self_outputs = self.self( 2025-12-04T09:35:42.4348478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4348594Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4348925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4349059Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4349357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4349442Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4349636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4349784Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4349789Z 2025-12-04T09:35:42.4349887Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4350230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4350297Z layer_outputs = layer_module( 2025-12-04T09:35:42.4350511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4350583Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4350849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4350959Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4351227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4351298Z self_outputs = self.self( 2025-12-04T09:35:42.4351566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4351673Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4352019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4352162Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4352166Z 2025-12-04T09:35:42.4352269Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4352611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4352682Z layer_outputs = layer_module( 2025-12-04T09:35:42.4352902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4352974Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4353257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4353327Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4353600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4353670Z self_outputs = self.self( 2025-12-04T09:35:42.4353939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4354048Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4354407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4354548Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4354551Z 2025-12-04T09:35:42.4354654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4354995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4355061Z layer_outputs = layer_module( 2025-12-04T09:35:42.4355277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4355347Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4355630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4355737Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4356003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4356074Z self_outputs = self.self( 2025-12-04T09:35:42.4356338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4356517Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4356521Z 2025-12-04T09:35:42.4356617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4356950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4357054Z layer_outputs = layer_module( 2025-12-04T09:35:42.4357270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4357342Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4357624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4357693Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4357978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4358086Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4358361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4358450Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4358453Z 2025-12-04T09:35:42.4358548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4358905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4358971Z layer_outputs = layer_module( 2025-12-04T09:35:42.4359186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4359268Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4359542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4359626Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4359881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4359956Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4360244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4360347Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4360622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4360707Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4360710Z 2025-12-04T09:35:42.4360807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4361158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4361225Z layer_outputs = layer_module( 2025-12-04T09:35:42.4361437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4361519Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4361827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4361914Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4362162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4362232Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4362505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4362603Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4362871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4363011Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4363220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4363296Z return self.act(input) 2025-12-04T09:35:42.4363300Z 2025-12-04T09:35:42.4363396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4363729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4363803Z layer_outputs = layer_module( 2025-12-04T09:35:42.4364012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4364093Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4364359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4364440Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4364697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4364767Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4365043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4365159Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4365424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4365508Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4365512Z 2025-12-04T09:35:42.4365607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4365950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4366023Z layer_outputs = layer_module( 2025-12-04T09:35:42.4366236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4366316Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4366588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4366660Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4366936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4367000Z self_outputs = self.self( 2025-12-04T09:35:42.4367274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4367356Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4367359Z 2025-12-04T09:35:42.4367482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4367824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4367890Z layer_outputs = layer_module( 2025-12-04T09:35:42.4368104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4368175Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4368440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4368518Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4368781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4368885Z self_outputs = self.self( 2025-12-04T09:35:42.4369154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4369251Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4369586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4369759Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4369763Z 2025-12-04T09:35:42.4369859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4370203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4370274Z layer_outputs = layer_module( 2025-12-04T09:35:42.4370492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4370564Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4370833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4370909Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4371178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4371251Z self_outputs = self.self( 2025-12-04T09:35:42.4371626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4371719Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4371728Z 2025-12-04T09:35:42.4371848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4372240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4372448Z layer_outputs = layer_module( 2025-12-04T09:35:42.4372689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4372769Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4373080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4373151Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4373415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4373487Z self_outputs = self.self( 2025-12-04T09:35:42.4373777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4373958Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4374335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4374527Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4374531Z 2025-12-04T09:35:42.4374647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4375030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4375111Z layer_outputs = layer_module( 2025-12-04T09:35:42.4375343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4375496Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4375813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4375892Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4376204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4376276Z self_outputs = self.self( 2025-12-04T09:35:42.4376577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4376688Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4377060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4377259Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4377266Z 2025-12-04T09:35:42.4377372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4377756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4377835Z layer_outputs = layer_module( 2025-12-04T09:35:42.4378070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4378152Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4378466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4378544Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4378855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4378930Z self_outputs = self.self( 2025-12-04T09:35:42.4379235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4379343Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4379706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4379877Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4379881Z 2025-12-04T09:35:42.4379956Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4380031Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4380111Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4380185Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4380282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4380653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4380721Z layer_outputs = layer_module( 2025-12-04T09:35:42.4380935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4381009Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4381271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4381349Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4381613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4381718Z self_outputs = self.self( 2025-12-04T09:35:42.4381986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4382088Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4382422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4382557Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4382875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4382965Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4382968Z 2025-12-04T09:35:42.4383041Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4383142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4383487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4383554Z layer_outputs = layer_module( 2025-12-04T09:35:42.4383770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4383843Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4384119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4384189Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4384456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4384530Z self_outputs = self.self( 2025-12-04T09:35:42.4384794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4384873Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4384879Z 2025-12-04T09:35:42.4384975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4385309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4385383Z layer_outputs = layer_module( 2025-12-04T09:35:42.4385590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4385661Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4385935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4386003Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4386280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4386372Z self_outputs = self.self( 2025-12-04T09:35:42.4386642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4386725Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4386729Z 2025-12-04T09:35:42.4386823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4387163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4387228Z layer_outputs = layer_module( 2025-12-04T09:35:42.4387435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4387514Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4387814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4387889Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4388152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4388215Z self_outputs = self.self( 2025-12-04T09:35:42.4388481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4388559Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4388563Z 2025-12-04T09:35:42.4388661Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4388998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4389066Z layer_outputs = layer_module( 2025-12-04T09:35:42.4389281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4389351Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4389617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4389692Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4389965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4390033Z self_outputs = self.self( 2025-12-04T09:35:42.4390296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4390406Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4390752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4390914Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4391104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4391197Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4391200Z 2025-12-04T09:35:42.4391295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4391632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4391698Z layer_outputs = layer_module( 2025-12-04T09:35:42.4391903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4391985Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4392686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4392774Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4393045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4393110Z self_outputs = self.self( 2025-12-04T09:35:42.4393386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4393497Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4393843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4394006Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4394311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4394406Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4394591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4394690Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4394693Z 2025-12-04T09:35:42.4394788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4395122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4395200Z layer_outputs = layer_module( 2025-12-04T09:35:42.4395407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4395483Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4395758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4395829Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4396101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4396167Z self_outputs = self.self( 2025-12-04T09:35:42.4396437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4396554Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4396894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4397050Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4397053Z 2025-12-04T09:35:42.4397153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4397492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4397565Z layer_outputs = layer_module( 2025-12-04T09:35:42.4397777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4397858Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4398129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4398199Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4398480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4398578Z self_outputs = self.self( 2025-12-04T09:35:42.4398848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4398965Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4399304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4399454Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4399457Z 2025-12-04T09:35:42.4399555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4399898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4400009Z layer_outputs = layer_module( 2025-12-04T09:35:42.4400227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4400311Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4400588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4400662Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4400948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4401014Z self_outputs = self.self( 2025-12-04T09:35:42.4401295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4401475Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4401481Z 2025-12-04T09:35:42.4401580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4401939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4402007Z layer_outputs = layer_module( 2025-12-04T09:35:42.4402230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4402306Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4402585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4402665Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4402950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4403067Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4403361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4403443Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4403446Z 2025-12-04T09:35:42.4403554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4403911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4403978Z layer_outputs = layer_module( 2025-12-04T09:35:42.4404201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4404275Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4404557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4404673Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4404930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4405010Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4405283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4405396Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4405664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4405743Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4405746Z 2025-12-04T09:35:42.4405850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4406223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4406297Z layer_outputs = layer_module( 2025-12-04T09:35:42.4406508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4406581Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4406856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4406936Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4407188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4407266Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4407541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4407655Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4407936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4408047Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4408268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4408339Z return self.act(input) 2025-12-04T09:35:42.4408342Z 2025-12-04T09:35:42.4408447Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4408787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4408857Z layer_outputs = layer_module( 2025-12-04T09:35:42.4409083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4409160Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4409433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4409521Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4409771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4409851Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4410125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4410244Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4410528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4410610Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4410643Z 2025-12-04T09:35:42.4410753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4411101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4411170Z layer_outputs = layer_module( 2025-12-04T09:35:42.4411414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4411646Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4411979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4412060Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4412405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4412491Z self_outputs = self.self( 2025-12-04T09:35:42.4412785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4412868Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4412880Z 2025-12-04T09:35:42.4412982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4413367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4413443Z layer_outputs = layer_module( 2025-12-04T09:35:42.4413656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4413732Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4414028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4414106Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4414393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4414461Z self_outputs = self.self( 2025-12-04T09:35:42.4414745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4414855Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4415195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4415384Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4415391Z 2025-12-04T09:35:42.4415491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4415853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4415931Z layer_outputs = layer_module( 2025-12-04T09:35:42.4416150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4416234Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4416515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4416588Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4416876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4416945Z self_outputs = self.self( 2025-12-04T09:35:42.4417253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4417348Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4417351Z 2025-12-04T09:35:42.4417458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4417821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4417893Z layer_outputs = layer_module( 2025-12-04T09:35:42.4418116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4418202Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4418486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4418597Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4418886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4418955Z self_outputs = self.self( 2025-12-04T09:35:42.4419239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4419338Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4419681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4419871Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4419875Z 2025-12-04T09:35:42.4419974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4420339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4420408Z layer_outputs = layer_module( 2025-12-04T09:35:42.4420624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4420706Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4420986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4421068Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4421347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4421413Z self_outputs = self.self( 2025-12-04T09:35:42.4421698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4421800Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4422149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4422327Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4422331Z 2025-12-04T09:35:42.4422431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4422785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4422852Z layer_outputs = layer_module( 2025-12-04T09:35:42.4423090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4423164Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4423464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4423544Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4423809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4423873Z self_outputs = self.self( 2025-12-04T09:35:42.4424146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4424238Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4424563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4424728Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4424762Z 2025-12-04T09:35:42.4424840Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4424925Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4424998Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4425076Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4425171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4425503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4425579Z layer_outputs = layer_module( 2025-12-04T09:35:42.4425786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4425859Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4426130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4426204Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4426479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4426545Z self_outputs = self.self( 2025-12-04T09:35:42.4426807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4426917Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4427238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4427381Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4427688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4427781Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4427786Z 2025-12-04T09:35:42.4427867Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4427960Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4428300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4428366Z layer_outputs = layer_module( 2025-12-04T09:35:42.4428575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4428654Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4428923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4428996Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4429307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4429376Z self_outputs = self.self( 2025-12-04T09:35:42.4429647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4429715Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4429718Z 2025-12-04T09:35:42.4429814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4430154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4430218Z layer_outputs = layer_module( 2025-12-04T09:35:42.4430436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4430542Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4430810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4430885Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4431151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4431213Z self_outputs = self.self( 2025-12-04T09:35:42.4431486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4431558Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4431561Z 2025-12-04T09:35:42.4431662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4431994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4432061Z layer_outputs = layer_module( 2025-12-04T09:35:42.4432278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4432349Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4432621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4432689Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4432954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4433025Z self_outputs = self.self( 2025-12-04T09:35:42.4433289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4433372Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4433378Z 2025-12-04T09:35:42.4433475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4433816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4433891Z layer_outputs = layer_module( 2025-12-04T09:35:42.4434099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4434170Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4434445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4434512Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4434784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4434851Z self_outputs = self.self( 2025-12-04T09:35:42.4435154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4435277Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4435608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4435779Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4435963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4436055Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4436058Z 2025-12-04T09:35:42.4436160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4436491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4436597Z layer_outputs = layer_module( 2025-12-04T09:35:42.4436820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4436892Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4437178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4437248Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4437524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4437595Z self_outputs = self.self( 2025-12-04T09:35:42.4437869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4437988Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4438337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4438466Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4438785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4438869Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4439065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4439157Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4439160Z 2025-12-04T09:35:42.4439257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4439616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4439683Z layer_outputs = layer_module( 2025-12-04T09:35:42.4439906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4439978Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4440257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4440333Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4440609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4440673Z self_outputs = self.self( 2025-12-04T09:35:42.4440955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4441065Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4441451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4441597Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4441600Z 2025-12-04T09:35:42.4441696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4442040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4442106Z layer_outputs = layer_module( 2025-12-04T09:35:42.4442315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4442436Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4442698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4442773Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4443030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4443100Z self_outputs = self.self( 2025-12-04T09:35:42.4443354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4443457Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4443785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4443922Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4443928Z 2025-12-04T09:35:42.4444028Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4444352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4444417Z layer_outputs = layer_module( 2025-12-04T09:35:42.4444625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4444695Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4444953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4445031Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4445285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4445356Z self_outputs = self.self( 2025-12-04T09:35:42.4445612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4445781Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4445791Z 2025-12-04T09:35:42.4445884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4446204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4446275Z layer_outputs = layer_module( 2025-12-04T09:35:42.4446475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4446544Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4446813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4446917Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4447185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4447288Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4447540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4447620Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4447624Z 2025-12-04T09:35:42.4447714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4448044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4448146Z layer_outputs = layer_module( 2025-12-04T09:35:42.4448357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4448437Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4448703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4448781Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4449036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4449108Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4449383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4449486Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4449754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4449843Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4449846Z 2025-12-04T09:35:42.4449941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4450280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4450346Z layer_outputs = layer_module( 2025-12-04T09:35:42.4450554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4450635Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4450901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4450989Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4451241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4451317Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4451694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4451807Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4452089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4452208Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4452426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4452505Z return self.act(input) 2025-12-04T09:35:42.4452509Z 2025-12-04T09:35:42.4452614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4452996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4453087Z layer_outputs = layer_module( 2025-12-04T09:35:42.4453301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4453384Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4453658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4453736Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4454002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4454073Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4454384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4454510Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4454785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4454867Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4454871Z 2025-12-04T09:35:42.4454964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4455306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4455379Z layer_outputs = layer_module( 2025-12-04T09:35:42.4455592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4455675Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4455953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4456024Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4456300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4456364Z self_outputs = self.self( 2025-12-04T09:35:42.4456638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4456716Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4456720Z 2025-12-04T09:35:42.4456815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4457162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4457231Z layer_outputs = layer_module( 2025-12-04T09:35:42.4457444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4457524Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4457796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4457872Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4458144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4458209Z self_outputs = self.self( 2025-12-04T09:35:42.4458486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4458583Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4458953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4459136Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4459140Z 2025-12-04T09:35:42.4459237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4459586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4459654Z layer_outputs = layer_module( 2025-12-04T09:35:42.4459876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4459949Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4460222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4460333Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4460611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4460679Z self_outputs = self.self( 2025-12-04T09:35:42.4460954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4461030Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4461033Z 2025-12-04T09:35:42.4461139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4461479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4461548Z layer_outputs = layer_module( 2025-12-04T09:35:42.4461773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4461850Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4462130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4462200Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4462470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4462544Z self_outputs = self.self( 2025-12-04T09:35:42.4462815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4462919Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4463247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4463429Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4463433Z 2025-12-04T09:35:42.4463553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4463892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4463965Z layer_outputs = layer_module( 2025-12-04T09:35:42.4464176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4464251Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4464529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4464598Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4464900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4464975Z self_outputs = self.self( 2025-12-04T09:35:42.4465244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4465348Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4465677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4465850Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4465854Z 2025-12-04T09:35:42.4465957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4466293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4466408Z layer_outputs = layer_module( 2025-12-04T09:35:42.4466629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4466705Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4466992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4467061Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4467344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4467410Z self_outputs = self.self( 2025-12-04T09:35:42.4467683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4467791Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4468131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4468313Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4468317Z 2025-12-04T09:35:42.4468395Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4468469Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4468549Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4468620Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4468717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4469078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4469147Z layer_outputs = layer_module( 2025-12-04T09:35:42.4469378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4469456Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4469735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4469812Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4470090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4470154Z self_outputs = self.self( 2025-12-04T09:35:42.4470438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4470540Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4470880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4471053Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4471370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4471470Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4471473Z 2025-12-04T09:35:42.4471546Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4471647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4471995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4472058Z layer_outputs = layer_module( 2025-12-04T09:35:42.4472417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4472535Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4472811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4472879Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4473137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4473207Z self_outputs = self.self( 2025-12-04T09:35:42.4473463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4473529Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4473533Z 2025-12-04T09:35:42.4473634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4473961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4474037Z layer_outputs = layer_module( 2025-12-04T09:35:42.4474242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4474313Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4474578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4474646Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4474910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4474972Z self_outputs = self.self( 2025-12-04T09:35:42.4475230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4475311Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4475315Z 2025-12-04T09:35:42.4475410Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4475737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4475802Z layer_outputs = layer_module( 2025-12-04T09:35:42.4476002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4476078Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4476336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4476403Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4476663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4476729Z self_outputs = self.self( 2025-12-04T09:35:42.4477052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4477133Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4477136Z 2025-12-04T09:35:42.4477460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4477804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4477869Z layer_outputs = layer_module( 2025-12-04T09:35:42.4478086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4478157Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4478424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4478549Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4478806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4478876Z self_outputs = self.self( 2025-12-04T09:35:42.4479136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4479243Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4479573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4479729Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4479905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4480004Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4480010Z 2025-12-04T09:35:42.4480102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4480428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4480492Z layer_outputs = layer_module( 2025-12-04T09:35:42.4480692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4480768Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4481023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4481099Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4481357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4481421Z self_outputs = self.self( 2025-12-04T09:35:42.4481683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4481789Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4482118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4482242Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4482534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4482625Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4482807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4482930Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4482940Z 2025-12-04T09:35:42.4483036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4483365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4483440Z layer_outputs = layer_module( 2025-12-04T09:35:42.4483645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4483716Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4483986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4484056Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4484403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4484469Z self_outputs = self.self( 2025-12-04T09:35:42.4484735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4484847Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4485182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4485334Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4485337Z 2025-12-04T09:35:42.4485433Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4485766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4485842Z layer_outputs = layer_module( 2025-12-04T09:35:42.4486056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4486135Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4486402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4486470Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4486742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4486805Z self_outputs = self.self( 2025-12-04T09:35:42.4487069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4487184Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4487524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4487671Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4487674Z 2025-12-04T09:35:42.4487770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4488103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4488176Z layer_outputs = layer_module( 2025-12-04T09:35:42.4488387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4488465Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4488737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4488836Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4489114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4489178Z self_outputs = self.self( 2025-12-04T09:35:42.4489447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4489623Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4489626Z 2025-12-04T09:35:42.4489720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4490057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4490165Z layer_outputs = layer_module( 2025-12-04T09:35:42.4490380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4490453Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4490717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4490792Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4491057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4491162Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4491445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4491590Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4491601Z 2025-12-04T09:35:42.4491716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4492076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4492146Z layer_outputs = layer_module( 2025-12-04T09:35:42.4492378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4492454Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4492814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4492893Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4493140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4493223Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4493495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4493604Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4493872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4493949Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4493952Z 2025-12-04T09:35:42.4494056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4494398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4494465Z layer_outputs = layer_module( 2025-12-04T09:35:42.4494698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4494774Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4495085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4495164Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4495413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4495492Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4495763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4495868Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4496135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4496279Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4496496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4496563Z return self.act(input) 2025-12-04T09:35:42.4496566Z 2025-12-04T09:35:42.4496667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4497006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4497072Z layer_outputs = layer_module( 2025-12-04T09:35:42.4497286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4497358Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4497625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4497712Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4497961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4498036Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4498304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4498417Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4498690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4498765Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4498769Z 2025-12-04T09:35:42.4498871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4499206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4499274Z layer_outputs = layer_module( 2025-12-04T09:35:42.4499491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4499561Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4499826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4499902Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4500168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4500239Z self_outputs = self.self( 2025-12-04T09:35:42.4500503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4500583Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4500586Z 2025-12-04T09:35:42.4500719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4501057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4501131Z layer_outputs = layer_module( 2025-12-04T09:35:42.4501341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4501412Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4501686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4501756Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4502027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4502127Z self_outputs = self.self( 2025-12-04T09:35:42.4502399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4502503Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4502827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4502998Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4503009Z 2025-12-04T09:35:42.4503105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4503437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4503512Z layer_outputs = layer_module( 2025-12-04T09:35:42.4503721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4503796Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4504075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4504143Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4504413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4504476Z self_outputs = self.self( 2025-12-04T09:35:42.4504740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4504818Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4504821Z 2025-12-04T09:35:42.4504920Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4505262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4505328Z layer_outputs = layer_module( 2025-12-04T09:35:42.4505536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4505614Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4505880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4505948Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4506222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4506284Z self_outputs = self.self( 2025-12-04T09:35:42.4506555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4506681Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4507006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4507186Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4507189Z 2025-12-04T09:35:42.4507285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4507624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4507691Z layer_outputs = layer_module( 2025-12-04T09:35:42.4507901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4508015Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4508282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4508360Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4508625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4508689Z self_outputs = self.self( 2025-12-04T09:35:42.4508958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4509053Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4509381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4509552Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4509555Z 2025-12-04T09:35:42.4509653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4509994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4510059Z layer_outputs = layer_module( 2025-12-04T09:35:42.4510271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4510342Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4510604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4510680Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4510937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4511002Z self_outputs = self.self( 2025-12-04T09:35:42.4511272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4511363Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4511689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4511858Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4511861Z 2025-12-04T09:35:42.4511936Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4512018Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4512090Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4512166Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4512263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4512634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4512709Z layer_outputs = layer_module( 2025-12-04T09:35:42.4512921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4512993Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4513268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4513336Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4513611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4513675Z self_outputs = self.self( 2025-12-04T09:35:42.4513969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4514083Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4514401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4514542Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4514847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4514936Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4514939Z 2025-12-04T09:35:42.4515020Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4515115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4515449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4515525Z layer_outputs = layer_module( 2025-12-04T09:35:42.4515733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4515812Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4516078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4516147Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4516427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4516492Z self_outputs = self.self( 2025-12-04T09:35:42.4516760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4516832Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4516835Z 2025-12-04T09:35:42.4516932Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4517269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4517336Z layer_outputs = layer_module( 2025-12-04T09:35:42.4517549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4517622Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4517885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4517961Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4518228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4518294Z self_outputs = self.self( 2025-12-04T09:35:42.4518595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4518669Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4518673Z 2025-12-04T09:35:42.4518777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4519114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4519181Z layer_outputs = layer_module( 2025-12-04T09:35:42.4519400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4519475Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4519783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4519853Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4520119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4520189Z self_outputs = self.self( 2025-12-04T09:35:42.4520451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4520527Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4520538Z 2025-12-04T09:35:42.4520630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4520966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4521038Z layer_outputs = layer_module( 2025-12-04T09:35:42.4521244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4521316Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4521589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4521655Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4521991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4522053Z self_outputs = self.self( 2025-12-04T09:35:42.4522306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4522420Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4522755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4522928Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4523112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4523206Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4523209Z 2025-12-04T09:35:42.4523313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4523646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4523719Z layer_outputs = layer_module( 2025-12-04T09:35:42.4523925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4524001Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4524347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4524420Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4524686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4524759Z self_outputs = self.self( 2025-12-04T09:35:42.4525026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4525140Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4525478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4525641Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4525955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4526041Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4526231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4526323Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4526327Z 2025-12-04T09:35:42.4526422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4526765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4526832Z layer_outputs = layer_module( 2025-12-04T09:35:42.4527048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4527123Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4527392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4527472Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4527739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4527803Z self_outputs = self.self( 2025-12-04T09:35:42.4528078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4528186Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4528525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4528673Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4528677Z 2025-12-04T09:35:42.4528775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4529113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4529179Z layer_outputs = layer_module( 2025-12-04T09:35:42.4529394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4529467Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4529733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4529811Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4530077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4530151Z self_outputs = self.self( 2025-12-04T09:35:42.4530447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4530555Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4530900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4531041Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4531044Z 2025-12-04T09:35:42.4531147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4531487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4531688Z layer_outputs = layer_module( 2025-12-04T09:35:42.4531943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4532025Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4532326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4532413Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4532718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4532794Z self_outputs = self.self( 2025-12-04T09:35:42.4533071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4533256Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4533263Z 2025-12-04T09:35:42.4533375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4533731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4533808Z layer_outputs = layer_module( 2025-12-04T09:35:42.4534026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4534111Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4534392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4534464Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4534748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4534857Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4535130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4535219Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4535222Z 2025-12-04T09:35:42.4535317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4535656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4535732Z layer_outputs = layer_module( 2025-12-04T09:35:42.4535946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4536025Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4536300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4536382Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4536697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4536769Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4537043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4537145Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4537412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4537500Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4537504Z 2025-12-04T09:35:42.4537600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4538007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4538077Z layer_outputs = layer_module( 2025-12-04T09:35:42.4538289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4538369Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4538641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4538721Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4538983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4539056Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4539339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4539445Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4539721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4539838Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4540057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4540129Z return self.act(input) 2025-12-04T09:35:42.4540133Z 2025-12-04T09:35:42.4540227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4540563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4540639Z layer_outputs = layer_module( 2025-12-04T09:35:42.4540849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4540926Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4541198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4541276Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4541533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4541603Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4541879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4542005Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4542279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4542368Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4542371Z 2025-12-04T09:35:42.4542499Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4542847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4542923Z layer_outputs = layer_module( 2025-12-04T09:35:42.4543143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4543227Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4543553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4543624Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4543908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4544007Z self_outputs = self.self( 2025-12-04T09:35:42.4544279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4544362Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4544366Z 2025-12-04T09:35:42.4544462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4544806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4544874Z layer_outputs = layer_module( 2025-12-04T09:35:42.4545083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4545168Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4545445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4545525Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4545795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4545860Z self_outputs = self.self( 2025-12-04T09:35:42.4546143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4546240Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4546580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4546760Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4546767Z 2025-12-04T09:35:42.4546863Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4547211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4547277Z layer_outputs = layer_module( 2025-12-04T09:35:42.4547492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4547565Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4547834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4547909Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4548176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4548243Z self_outputs = self.self( 2025-12-04T09:35:42.4548548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4548626Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4548630Z 2025-12-04T09:35:42.4548734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4549080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4549145Z layer_outputs = layer_module( 2025-12-04T09:35:42.4549360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4549430Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4549699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4549799Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4550065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4550138Z self_outputs = self.self( 2025-12-04T09:35:42.4550403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4550496Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4550822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4550991Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4550994Z 2025-12-04T09:35:42.4551094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4551426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4551500Z layer_outputs = layer_module( 2025-12-04T09:35:42.4551712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4551781Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4552049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4552118Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4552377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4552447Z self_outputs = self.self( 2025-12-04T09:35:42.4552703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4552802Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4553123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4553291Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4553294Z 2025-12-04T09:35:42.4553396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4553723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4553794Z layer_outputs = layer_module( 2025-12-04T09:35:42.4553999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4554071Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4554382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4554453Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4554726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4554793Z self_outputs = self.self( 2025-12-04T09:35:42.4555057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4555156Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4555479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4555648Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4555698Z 2025-12-04T09:35:42.4555777Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4555854Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4555932Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4556002Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4556099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4556439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4556504Z layer_outputs = layer_module( 2025-12-04T09:35:42.4556716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4556794Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4557064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4557144Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4557416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4557482Z self_outputs = self.self( 2025-12-04T09:35:42.4557759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4557865Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4558210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4558347Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4558662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4558766Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4558769Z 2025-12-04T09:35:42.4558846Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4558949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4559292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4559359Z layer_outputs = layer_module( 2025-12-04T09:35:42.4559580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4559653Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4559927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4560005Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4560277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4560385Z self_outputs = self.self( 2025-12-04T09:35:42.4560657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4560728Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4560732Z 2025-12-04T09:35:42.4560837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4561179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4561253Z layer_outputs = layer_module( 2025-12-04T09:35:42.4561470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4561577Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4561861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4561933Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4562215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4562279Z self_outputs = self.self( 2025-12-04T09:35:42.4562552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4562633Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4562636Z 2025-12-04T09:35:42.4562734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4563081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4563158Z layer_outputs = layer_module( 2025-12-04T09:35:42.4563376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4563453Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4563729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4563800Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4564079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4564141Z self_outputs = self.self( 2025-12-04T09:35:42.4564421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4564500Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4564507Z 2025-12-04T09:35:42.4564604Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4564955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4565020Z layer_outputs = layer_module( 2025-12-04T09:35:42.4565238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4565311Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4565585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4565659Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4565935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4566002Z self_outputs = self.self( 2025-12-04T09:35:42.4566320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4566440Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4566798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4566967Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4567159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4567262Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4567266Z 2025-12-04T09:35:42.4567366Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4567720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4567846Z layer_outputs = layer_module( 2025-12-04T09:35:42.4568070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4568155Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4568440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4568518Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4568799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4568867Z self_outputs = self.self( 2025-12-04T09:35:42.4569155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4569273Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4569631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4569773Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4570101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4570196Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4570393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4570494Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4570498Z 2025-12-04T09:35:42.4570611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4570997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4571086Z layer_outputs = layer_module( 2025-12-04T09:35:42.4571322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4571402Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4571809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4571898Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4572226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4572441Z self_outputs = self.self( 2025-12-04T09:35:42.4572754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4572893Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4573331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4573481Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4573492Z 2025-12-04T09:35:42.4573591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4573932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4574009Z layer_outputs = layer_module( 2025-12-04T09:35:42.4574221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4574305Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4574647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4574725Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4575006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4575075Z self_outputs = self.self( 2025-12-04T09:35:42.4575348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4575468Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4575823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4575975Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4575981Z 2025-12-04T09:35:42.4576081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4576443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4576518Z layer_outputs = layer_module( 2025-12-04T09:35:42.4576734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4576817Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4577106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4577181Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4577476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4577545Z self_outputs = self.self( 2025-12-04T09:35:42.4577829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4578022Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4578026Z 2025-12-04T09:35:42.4578128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4578497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4578565Z layer_outputs = layer_module( 2025-12-04T09:35:42.4578787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4578871Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4579154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4579237Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4579555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4579670Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4579957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4580039Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4580042Z 2025-12-04T09:35:42.4580150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4580502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4580572Z layer_outputs = layer_module( 2025-12-04T09:35:42.4580839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4580918Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4581195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4581286Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4581550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4581632Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4581911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4582018Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4582300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4582383Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4582389Z 2025-12-04T09:35:42.4582494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4582841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4582910Z layer_outputs = layer_module( 2025-12-04T09:35:42.4583133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4583205Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4583488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4583568Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4583832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4583914Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4584196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4584300Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4584581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4584691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4584913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4584981Z return self.act(input) 2025-12-04T09:35:42.4584984Z 2025-12-04T09:35:42.4585082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4585474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4585545Z layer_outputs = layer_module( 2025-12-04T09:35:42.4585766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4585843Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4586120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4586207Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4586468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4586540Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4586822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4586977Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4587261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4587340Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4587344Z 2025-12-04T09:35:42.4587444Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4587800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4587870Z layer_outputs = layer_module( 2025-12-04T09:35:42.4588095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4588167Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4588447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4588530Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4588810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4588885Z self_outputs = self.self( 2025-12-04T09:35:42.4589169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4589245Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4589249Z 2025-12-04T09:35:42.4589355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4589692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4589772Z layer_outputs = layer_module( 2025-12-04T09:35:42.4589984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4590058Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4590335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4590407Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4590679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4590755Z self_outputs = self.self( 2025-12-04T09:35:42.4591022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4591126Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4591490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4591672Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4591675Z 2025-12-04T09:35:42.4591781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4592123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4592198Z layer_outputs = layer_module( 2025-12-04T09:35:42.4592411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4592483Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4592761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4592865Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4593150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4593215Z self_outputs = self.self( 2025-12-04T09:35:42.4593486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4593568Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4593571Z 2025-12-04T09:35:42.4593670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4594014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4594091Z layer_outputs = layer_module( 2025-12-04T09:35:42.4594305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4594389Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4594667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4594738Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4595022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4595087Z self_outputs = self.self( 2025-12-04T09:35:42.4595366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4595465Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4595800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4595990Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4595995Z 2025-12-04T09:35:42.4596093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4596442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4596510Z layer_outputs = layer_module( 2025-12-04T09:35:42.4596726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4596808Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4597084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4597154Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4597441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4597537Z self_outputs = self.self( 2025-12-04T09:35:42.4597817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4597913Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4598244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4598427Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4598430Z 2025-12-04T09:35:42.4598527Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4598877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4598977Z layer_outputs = layer_module( 2025-12-04T09:35:42.4599198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4599281Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4599551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4599632Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4599905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4599969Z self_outputs = self.self( 2025-12-04T09:35:42.4600243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4600340Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4600675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4600848Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4600851Z 2025-12-04T09:35:42.4600927Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4601010Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4601082Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4601152Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4601258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4601599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4601672Z layer_outputs = layer_module( 2025-12-04T09:35:42.4601886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4601963Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4602246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4602317Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4602588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4602659Z self_outputs = self.self( 2025-12-04T09:35:42.4602928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4603039Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4603368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4603551Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4603871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4603964Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4603967Z 2025-12-04T09:35:42.4604047Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4604145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4604485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4604560Z layer_outputs = layer_module( 2025-12-04T09:35:42.4604773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4604887Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4605162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4605234Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4605551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4605616Z self_outputs = self.self( 2025-12-04T09:35:42.4605888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4605964Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4605967Z 2025-12-04T09:35:42.4606064Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4606413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4606483Z layer_outputs = layer_module( 2025-12-04T09:35:42.4606696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4606778Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4607048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4607124Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4607395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4607461Z self_outputs = self.self( 2025-12-04T09:35:42.4607734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4607814Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4607817Z 2025-12-04T09:35:42.4607922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4608265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4608332Z layer_outputs = layer_module( 2025-12-04T09:35:42.4608549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4608622Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4608891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4608969Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4609243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4609318Z self_outputs = self.self( 2025-12-04T09:35:42.4609623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4609707Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4609711Z 2025-12-04T09:35:42.4609826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4610185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4610261Z layer_outputs = layer_module( 2025-12-04T09:35:42.4610470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4610541Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4610818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4610922Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4611202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4611267Z self_outputs = self.self( 2025-12-04T09:35:42.4611636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4611780Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4612169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4612356Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4612578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4612690Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4612694Z 2025-12-04T09:35:42.4612815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4613201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4613269Z layer_outputs = layer_module( 2025-12-04T09:35:42.4613492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4613567Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4613850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4613923Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4614197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4614278Z self_outputs = self.self( 2025-12-04T09:35:42.4614556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4614675Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4615015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4615147Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4615464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4615553Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4615741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4615845Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4615889Z 2025-12-04T09:35:42.4615991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4616337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4616404Z layer_outputs = layer_module( 2025-12-04T09:35:42.4616617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4616697Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4616968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4617046Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4617348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4617417Z self_outputs = self.self( 2025-12-04T09:35:42.4617691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4617798Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4618139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4618283Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4618286Z 2025-12-04T09:35:42.4618382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4618727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4618797Z layer_outputs = layer_module( 2025-12-04T09:35:42.4619016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4619089Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4619355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4619433Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4619706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4619770Z self_outputs = self.self( 2025-12-04T09:35:42.4620041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4620147Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4620498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4620642Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4620645Z 2025-12-04T09:35:42.4620742Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4621085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4621152Z layer_outputs = layer_module( 2025-12-04T09:35:42.4621378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4621449Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4621709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4621791Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4622083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4622157Z self_outputs = self.self( 2025-12-04T09:35:42.4622420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4622593Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4622596Z 2025-12-04T09:35:42.4622696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4623028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4623132Z layer_outputs = layer_module( 2025-12-04T09:35:42.4623344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4623418Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4623694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4623764Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4624031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4624145Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4624412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4624497Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4624504Z 2025-12-04T09:35:42.4624598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4624936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4625012Z layer_outputs = layer_module( 2025-12-04T09:35:42.4625224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4625299Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4625567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4625645Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4625903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4625975Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4626247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4626359Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4626628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4626711Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4626715Z 2025-12-04T09:35:42.4626808Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4627145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4627219Z layer_outputs = layer_module( 2025-12-04T09:35:42.4627429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4627508Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4627828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4627908Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4628164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4628235Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4628510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4628610Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4628883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4628997Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4629237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4629305Z return self.act(input) 2025-12-04T09:35:42.4629315Z 2025-12-04T09:35:42.4629409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4629737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4629809Z layer_outputs = layer_module( 2025-12-04T09:35:42.4630017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4630086Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4630360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4630439Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4630693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4630761Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4631028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4631148Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4631413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4631495Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4631498Z 2025-12-04T09:35:42.4631591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4631922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4631998Z layer_outputs = layer_module( 2025-12-04T09:35:42.4632207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4632278Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4632549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4632619Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4632886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4632949Z self_outputs = self.self( 2025-12-04T09:35:42.4633212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T09:35:42.4633298Z query_vectors = self.query(hidden_states) 2025-12-04T09:35:42.4633301Z 2025-12-04T09:35:42.4633394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4633787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4633857Z layer_outputs = layer_module( 2025-12-04T09:35:42.4634064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4634140Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4634403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4634477Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4634737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4634836Z self_outputs = self.self( 2025-12-04T09:35:42.4635108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4635204Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4635525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4635703Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4635706Z 2025-12-04T09:35:42.4635798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4636140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4636206Z layer_outputs = layer_module( 2025-12-04T09:35:42.4636421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4636502Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4636773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4636849Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4637122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4637186Z self_outputs = self.self( 2025-12-04T09:35:42.4637461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T09:35:42.4637534Z key_vectors = self.key(hidden_states) 2025-12-04T09:35:42.4637538Z 2025-12-04T09:35:42.4637641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4637986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4638053Z layer_outputs = layer_module( 2025-12-04T09:35:42.4638271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4638345Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4638615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4638694Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4638964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4639037Z self_outputs = self.self( 2025-12-04T09:35:42.4639303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4639403Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4639768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4639949Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4639953Z 2025-12-04T09:35:42.4640061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4640402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4640474Z layer_outputs = layer_module( 2025-12-04T09:35:42.4640696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4640801Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4641081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4641151Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4641423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4641496Z self_outputs = self.self( 2025-12-04T09:35:42.4641765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4641861Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4642198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4642372Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4642375Z 2025-12-04T09:35:42.4642483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4642820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4642885Z layer_outputs = layer_module( 2025-12-04T09:35:42.4643104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4643177Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4643455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4643525Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4643797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4643874Z self_outputs = self.self( 2025-12-04T09:35:42.4644147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T09:35:42.4644249Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4644578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4644748Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T09:35:42.4644751Z 2025-12-04T09:35:42.4644835Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4644911Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4644991Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4645063Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4645163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4645539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4645610Z layer_outputs = layer_module( 2025-12-04T09:35:42.4645825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4645908Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4646183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4646261Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4646532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4646598Z self_outputs = self.self( 2025-12-04T09:35:42.4646913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T09:35:42.4647021Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T09:35:42.4647352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T09:35:42.4647495Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T09:35:42.4647807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T09:35:42.4647908Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T09:35:42.4647911Z 2025-12-04T09:35:42.4647985Z cudagraph partition due to non gpu ops 2025-12-04T09:35:42.4648082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4648430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4648504Z layer_outputs = layer_module( 2025-12-04T09:35:42.4648728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4648804Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4649085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4649165Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4649445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4649518Z self_outputs = self.self( 2025-12-04T09:35:42.4649796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T09:35:42.4649869Z attn_scores += diagonal_mask 2025-12-04T09:35:42.4649872Z 2025-12-04T09:35:42.4649982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4650332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4650406Z layer_outputs = layer_module( 2025-12-04T09:35:42.4650623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4650698Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4650981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4651053Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4651388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4651472Z self_outputs = self.self( 2025-12-04T09:35:42.4651903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T09:35:42.4652003Z attn_probs = nn.functional.softmax( 2025-12-04T09:35:42.4652008Z 2025-12-04T09:35:42.4652120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4652509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4652589Z layer_outputs = layer_module( 2025-12-04T09:35:42.4652819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4652908Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4653203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4653313Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4653620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4653692Z self_outputs = self.self( 2025-12-04T09:35:42.4653983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T09:35:42.4654078Z value_vectors = self.value(hidden_states) 2025-12-04T09:35:42.4654082Z 2025-12-04T09:35:42.4654189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4654566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4654638Z layer_outputs = layer_module( 2025-12-04T09:35:42.4654874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4654964Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4655260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4655343Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4655643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4655710Z self_outputs = self.self( 2025-12-04T09:35:42.4655993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4656110Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4656464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4656642Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T09:35:42.4656848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4656955Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4656958Z 2025-12-04T09:35:42.4657062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4657433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4657516Z layer_outputs = layer_module( 2025-12-04T09:35:42.4657747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4657831Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4658168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4658247Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4658552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4658624Z self_outputs = self.self( 2025-12-04T09:35:42.4658927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4659046Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4659421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4659570Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T09:35:42.4659950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T09:35:42.4660055Z chunked_hidden_states = nn.functional.pad( 2025-12-04T09:35:42.4660259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T09:35:42.4660362Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T09:35:42.4660366Z 2025-12-04T09:35:42.4660480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4660857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4660937Z layer_outputs = layer_module( 2025-12-04T09:35:42.4661171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4661253Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4661560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4661638Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4661936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4662015Z self_outputs = self.self( 2025-12-04T09:35:42.4662309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4662432Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4662804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4662972Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4662976Z 2025-12-04T09:35:42.4663092Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4663463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4663543Z layer_outputs = layer_module( 2025-12-04T09:35:42.4663778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4663857Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4664164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4664242Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4664545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4664619Z self_outputs = self.self( 2025-12-04T09:35:42.4664952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T09:35:42.4665083Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T09:35:42.4665454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T09:35:42.4665609Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T09:35:42.4665629Z 2025-12-04T09:35:42.4665727Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4666070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4666179Z layer_outputs = layer_module( 2025-12-04T09:35:42.4666394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4666471Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4666751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4666820Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4667097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T09:35:42.4667162Z self_outputs = self.self( 2025-12-04T09:35:42.4667432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T09:35:42.4667616Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T09:35:42.4667623Z 2025-12-04T09:35:42.4667720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4668071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4668138Z layer_outputs = layer_module( 2025-12-04T09:35:42.4668352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4668431Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4668703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T09:35:42.4668773Z self_attn_outputs = self.attention( 2025-12-04T09:35:42.4669051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T09:35:42.4669161Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:35:42.4669442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T09:35:42.4669520Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4669524Z 2025-12-04T09:35:42.4669619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4669970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4670037Z layer_outputs = layer_module( 2025-12-04T09:35:42.4670257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4670331Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4670604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4670697Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4670987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4671070Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4671344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4671447Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4671721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T09:35:42.4671800Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4671804Z 2025-12-04T09:35:42.4671909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4672248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4672576Z layer_outputs = layer_module( 2025-12-04T09:35:42.4672807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4672885Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4673163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4673252Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4673510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4673593Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4673877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T09:35:42.4673987Z intermediate_output = self.intermediate(attn_output) 2025-12-04T09:35:42.4674281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T09:35:42.4674391Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:35:42.4674608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:35:42.4674678Z return self.act(input) 2025-12-04T09:35:42.4674682Z 2025-12-04T09:35:42.4674781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:35:42.4675129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T09:35:42.4675198Z layer_outputs = layer_module( 2025-12-04T09:35:42.4675415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:35:42.4675500Z return super().__call__(*args, **kwargs) 2025-12-04T09:35:42.4675774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T09:35:42.4675864Z layer_output = apply_chunking_to_forward( 2025-12-04T09:35:42.4676116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:35:42.4676187Z return forward_fn(*input_tensors) 2025-12-04T09:35:42.4676471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T09:35:42.4676587Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T09:35:42.4676871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T09:35:42.4676951Z hidden_states = self.dense(hidden_states) 2025-12-04T09:35:42.4676954Z 2025-12-04T09:36:48.1718944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:36:48.1719935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1715, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T09:36:48.1720470Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T09:36:48.1720916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1332, in forward 2025-12-04T09:36:48.1721344Z x = self.dense(features) 2025-12-04T09:36:48.1721461Z 2025-12-04T09:36:48.1721578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:36:48.1722092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1715, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T09:36:48.1722752Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T09:36:48.1723202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1337, in forward 2025-12-04T09:36:48.1723625Z x = self.decoder(x) 2025-12-04T09:36:48.1723732Z 2025-12-04T09:36:48.1723841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:36:48.1724353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1722, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T09:36:48.1724960Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:36:48.1725203Z 2025-12-04T09:36:49.5407142Z Compilation time (from dynamo_timed): 98.657265066 2025-12-04T09:36:49.5640028Z pass 2025-12-04T09:36:49.5641884Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:36:49.5642840Z TIMING: gc:0.00476 entire_frame_compile:98.65727 _recursive_pre_grad_passes:0.01813 _recursive_joint_graph_passes:0.94715 _recursive_post_grad_passes:1.51542 async_compile.wait:2.94289 code_gen:75.98919 inductor_compile:83.0593 backend_compile:93.18077 total_wall_time:98.65727 2025-12-04T09:36:49.5643859Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:39711 | FakeTensor.__torch_dispatch__:16210 | ProxyTorchDispatchMode.__torch_dispatch__:10021 2025-12-04T09:36:49.5644360Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-12-04T09:36:53.1303995Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:36:53.1304825Z import pynvml # type: ignore[import] 2025-12-04T09:36:56.4282056Z 2025-12-04T09:36:58.7354571Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:36:58.7356662Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:36:58.7375144Z cpu eval BartForCausalLM 2025-12-04T09:37:00.4463672Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:37:01.1117777Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:37:01.7266164Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:37:08.5285856Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5286491Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5286876Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5287763Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5288274Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5288636Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5289030Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5289757Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5290100Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5290744Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5290984Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5291224Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5291724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5292180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5292548Z res = mod(**inputs) 2025-12-04T09:37:08.5292998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5293432Z outputs = self.model.decoder( 2025-12-04T09:37:08.5293821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5294317Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5294684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5295062Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5295457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5295829Z return func(*args, **kwargs) 2025-12-04T09:37:08.5296204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5296630Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5297038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5297428Z return func(*args, **kwargs) 2025-12-04T09:37:08.5297801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5298268Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5298479Z 2025-12-04T09:37:08.5298589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5299032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5299368Z res = mod(**inputs) 2025-12-04T09:37:08.5299741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5300127Z outputs = self.model.decoder( 2025-12-04T09:37:08.5300565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5300956Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5301304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5301677Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5302060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5302435Z return func(*args, **kwargs) 2025-12-04T09:37:08.5302796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5303204Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5303614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5303982Z return func(*args, **kwargs) 2025-12-04T09:37:08.5304355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5304750Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5304892Z 2025-12-04T09:37:08.5305004Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5305404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5305736Z res = mod(**inputs) 2025-12-04T09:37:08.5306098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5306489Z outputs = self.model.decoder( 2025-12-04T09:37:08.5306860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5307242Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5307593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5307956Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5308337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5308799Z return func(*args, **kwargs) 2025-12-04T09:37:08.5309398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5309801Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5310203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5310575Z return func(*args, **kwargs) 2025-12-04T09:37:08.5310943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5311340Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5311485Z 2025-12-04T09:37:08.5311561Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5311794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5312140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5312470Z res = mod(**inputs) 2025-12-04T09:37:08.5312861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5313255Z outputs = self.model.decoder( 2025-12-04T09:37:08.5313630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5314027Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5314378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5314730Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5315101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5315493Z return func(*args, **kwargs) 2025-12-04T09:37:08.5315862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5316268Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5316653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5317016Z return func(*args, **kwargs) 2025-12-04T09:37:08.5317377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5317773Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5318211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5318692Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5318870Z 2025-12-04T09:37:08.5318978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5319325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5319675Z res = mod(**inputs) 2025-12-04T09:37:08.5320029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5320410Z outputs = self.model.decoder( 2025-12-04T09:37:08.5320768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5321141Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5321481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5321827Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5322195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5322598Z return func(*args, **kwargs) 2025-12-04T09:37:08.5322966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5323355Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5323747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5324109Z return func(*args, **kwargs) 2025-12-04T09:37:08.5324465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5324842Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5324980Z 2025-12-04T09:37:08.5325081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5325428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5325739Z res = mod(**inputs) 2025-12-04T09:37:08.5326101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5326492Z outputs = self.model.decoder( 2025-12-04T09:37:08.5326869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5327245Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5327597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5327965Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5328335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5328703Z return func(*args, **kwargs) 2025-12-04T09:37:08.5329068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5329504Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5329678Z 2025-12-04T09:37:08.5329785Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5330143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5330462Z res = mod(**inputs) 2025-12-04T09:37:08.5330829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5331229Z outputs = self.model.decoder( 2025-12-04T09:37:08.5331823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5332253Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5332627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5333081Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5333469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5333889Z return func(*args, **kwargs) 2025-12-04T09:37:08.5334253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5334682Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5335073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5335416Z return self.act(input) 2025-12-04T09:37:08.5335533Z 2025-12-04T09:37:08.5335638Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5336001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5336321Z res = mod(**inputs) 2025-12-04T09:37:08.5336671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5337969Z outputs = self.model.decoder( 2025-12-04T09:37:08.5338350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5338733Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5339079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5339445Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5339819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5340180Z return func(*args, **kwargs) 2025-12-04T09:37:08.5340544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5340930Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5341071Z 2025-12-04T09:37:08.5341178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5341528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5341848Z res = mod(**inputs) 2025-12-04T09:37:08.5342199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5342579Z outputs = self.model.decoder( 2025-12-04T09:37:08.5342941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5343311Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5343651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5343992Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5344355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5344718Z return func(*args, **kwargs) 2025-12-04T09:37:08.5345076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5345460Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5345845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5346205Z return func(*args, **kwargs) 2025-12-04T09:37:08.5346609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5347058Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5347262Z 2025-12-04T09:37:08.5347361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5347711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5348016Z res = mod(**inputs) 2025-12-04T09:37:08.5348394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5348772Z outputs = self.model.decoder( 2025-12-04T09:37:08.5349132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5349506Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5349849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5350205Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5350567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5350933Z return func(*args, **kwargs) 2025-12-04T09:37:08.5351326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5351726Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5352115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5352476Z return func(*args, **kwargs) 2025-12-04T09:37:08.5352838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5353271Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5353409Z 2025-12-04T09:37:08.5353509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5353859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5354174Z res = mod(**inputs) 2025-12-04T09:37:08.5354513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5354888Z outputs = self.model.decoder( 2025-12-04T09:37:08.5355258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5355621Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5355960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5356313Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5356680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5357032Z return func(*args, **kwargs) 2025-12-04T09:37:08.5357398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5357782Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5358171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5358521Z return func(*args, **kwargs) 2025-12-04T09:37:08.5358879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5359263Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5359398Z 2025-12-04T09:37:08.5359475Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5359708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5360052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5360364Z res = mod(**inputs) 2025-12-04T09:37:08.5360701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5361075Z outputs = self.model.decoder( 2025-12-04T09:37:08.5361475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5361845Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5362176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5362517Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5362876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5363218Z return func(*args, **kwargs) 2025-12-04T09:37:08.5363565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5363953Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5364329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5364720Z return func(*args, **kwargs) 2025-12-04T09:37:08.5365082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5365480Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5365911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5366391Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5366579Z 2025-12-04T09:37:08.5366683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5367035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5367341Z res = mod(**inputs) 2025-12-04T09:37:08.5367689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5368076Z outputs = self.model.decoder( 2025-12-04T09:37:08.5368450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5368833Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5369183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5369546Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5369918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5370292Z return func(*args, **kwargs) 2025-12-04T09:37:08.5370660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5371069Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5371555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5371975Z return func(*args, **kwargs) 2025-12-04T09:37:08.5372670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5373111Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5373259Z 2025-12-04T09:37:08.5373364Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5373727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5374050Z res = mod(**inputs) 2025-12-04T09:37:08.5374404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5374793Z outputs = self.model.decoder( 2025-12-04T09:37:08.5375172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5375597Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5376032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5376395Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5376774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5377137Z return func(*args, **kwargs) 2025-12-04T09:37:08.5377513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5377939Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5378110Z 2025-12-04T09:37:08.5378222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5378569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5378944Z res = mod(**inputs) 2025-12-04T09:37:08.5379300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5379676Z outputs = self.model.decoder( 2025-12-04T09:37:08.5380050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5380432Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5380776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5381131Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5381508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5381877Z return func(*args, **kwargs) 2025-12-04T09:37:08.5382236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5382663Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5383056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5383399Z return self.act(input) 2025-12-04T09:37:08.5383509Z 2025-12-04T09:37:08.5383612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5383968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5384286Z res = mod(**inputs) 2025-12-04T09:37:08.5384640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5385025Z outputs = self.model.decoder( 2025-12-04T09:37:08.5385397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5385782Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5386125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5386482Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5386858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5387228Z return func(*args, **kwargs) 2025-12-04T09:37:08.5387585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5387972Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5388116Z 2025-12-04T09:37:08.5388222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5388559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5388871Z res = mod(**inputs) 2025-12-04T09:37:08.5389218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5389630Z outputs = self.model.decoder( 2025-12-04T09:37:08.5390000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5390388Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5390738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5391099Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5391471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5391843Z return func(*args, **kwargs) 2025-12-04T09:37:08.5392214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5392635Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5393020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5393377Z return func(*args, **kwargs) 2025-12-04T09:37:08.5393728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5394162Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5394364Z 2025-12-04T09:37:08.5394462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5394804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5395102Z res = mod(**inputs) 2025-12-04T09:37:08.5395443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5395818Z outputs = self.model.decoder( 2025-12-04T09:37:08.5396183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5396546Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5396883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5397241Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5397605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5397960Z return func(*args, **kwargs) 2025-12-04T09:37:08.5398314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5398708Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5399086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5399455Z return func(*args, **kwargs) 2025-12-04T09:37:08.5399814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5400189Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5400323Z 2025-12-04T09:37:08.5400423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5400767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5401072Z res = mod(**inputs) 2025-12-04T09:37:08.5401404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5401806Z outputs = self.model.decoder( 2025-12-04T09:37:08.5402173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5402533Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5402882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5403230Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5403588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5403939Z return func(*args, **kwargs) 2025-12-04T09:37:08.5404282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5404674Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5405061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5405412Z return func(*args, **kwargs) 2025-12-04T09:37:08.5405775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5406196Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5406334Z 2025-12-04T09:37:08.5406422Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5406648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5406994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5407309Z res = mod(**inputs) 2025-12-04T09:37:08.5407650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5408025Z outputs = self.model.decoder( 2025-12-04T09:37:08.5408391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5408762Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5409094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5409455Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5409827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5410193Z return func(*args, **kwargs) 2025-12-04T09:37:08.5410546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5410952Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5411349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5411802Z return func(*args, **kwargs) 2025-12-04T09:37:08.5412212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5412697Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5413177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5413681Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5413870Z 2025-12-04T09:37:08.5413971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5414334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5414658Z res = mod(**inputs) 2025-12-04T09:37:08.5415009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5415396Z outputs = self.model.decoder( 2025-12-04T09:37:08.5415779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5416163Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5416516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5416927Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5417317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5417679Z return func(*args, **kwargs) 2025-12-04T09:37:08.5418041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5418452Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5418844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5419225Z return func(*args, **kwargs) 2025-12-04T09:37:08.5419607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5420022Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5420153Z 2025-12-04T09:37:08.5420258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5420606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5420928Z res = mod(**inputs) 2025-12-04T09:37:08.5421269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5421643Z outputs = self.model.decoder( 2025-12-04T09:37:08.5422008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5422381Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5422715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5423076Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5423452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5423816Z return func(*args, **kwargs) 2025-12-04T09:37:08.5424162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5424577Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5424743Z 2025-12-04T09:37:08.5424848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5425186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5425497Z res = mod(**inputs) 2025-12-04T09:37:08.5425840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5426211Z outputs = self.model.decoder( 2025-12-04T09:37:08.5426571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5426941Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5427281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5427622Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5427987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5428344Z return func(*args, **kwargs) 2025-12-04T09:37:08.5428701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5429107Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5429481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5429824Z return self.act(input) 2025-12-04T09:37:08.5429929Z 2025-12-04T09:37:08.5430035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5430408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5430723Z res = mod(**inputs) 2025-12-04T09:37:08.5431070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5431437Z outputs = self.model.decoder( 2025-12-04T09:37:08.5431809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5432181Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5432519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5432865Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5433275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5433643Z return func(*args, **kwargs) 2025-12-04T09:37:08.5433996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5434390Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5434531Z 2025-12-04T09:37:08.5434631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5434977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5435283Z res = mod(**inputs) 2025-12-04T09:37:08.5435629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5436002Z outputs = self.model.decoder( 2025-12-04T09:37:08.5436356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5436714Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5437043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5437387Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5437738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5438088Z return func(*args, **kwargs) 2025-12-04T09:37:08.5438443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5438833Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5439206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5439555Z return func(*args, **kwargs) 2025-12-04T09:37:08.5439910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5440340Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5440538Z 2025-12-04T09:37:08.5440637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5440976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5441282Z res = mod(**inputs) 2025-12-04T09:37:08.5441609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5441975Z outputs = self.model.decoder( 2025-12-04T09:37:08.5442335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5442705Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5443044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5443430Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5443811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5444157Z return func(*args, **kwargs) 2025-12-04T09:37:08.5444510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5444909Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5445299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5445651Z return func(*args, **kwargs) 2025-12-04T09:37:08.5446010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5446432Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5446564Z 2025-12-04T09:37:08.5446669Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5447010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5447320Z res = mod(**inputs) 2025-12-04T09:37:08.5447666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5448067Z outputs = self.model.decoder( 2025-12-04T09:37:08.5448445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5448835Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5449174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5449523Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5449900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5450267Z return func(*args, **kwargs) 2025-12-04T09:37:08.5450625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5451035Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5451530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5451945Z return func(*args, **kwargs) 2025-12-04T09:37:08.5452342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5452766Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5452919Z 2025-12-04T09:37:08.5453009Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5453240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5453612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5453956Z res = mod(**inputs) 2025-12-04T09:37:08.5454340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5454743Z outputs = self.model.decoder( 2025-12-04T09:37:08.5455148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5455555Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5455927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5456314Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5456726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5457132Z return func(*args, **kwargs) 2025-12-04T09:37:08.5457556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5457998Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5458432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5458833Z return func(*args, **kwargs) 2025-12-04T09:37:08.5459212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5459644Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5460115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5460632Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5460849Z 2025-12-04T09:37:08.5460948Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5461298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5461609Z res = mod(**inputs) 2025-12-04T09:37:08.5461950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5462322Z outputs = self.model.decoder( 2025-12-04T09:37:08.5462689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5463063Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5463394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5463751Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5464123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5464484Z return func(*args, **kwargs) 2025-12-04T09:37:08.5464849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5465246Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5465635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5465992Z return func(*args, **kwargs) 2025-12-04T09:37:08.5466354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5466740Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5466873Z 2025-12-04T09:37:08.5466979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5467323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5467640Z res = mod(**inputs) 2025-12-04T09:37:08.5467992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5468363Z outputs = self.model.decoder( 2025-12-04T09:37:08.5468734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5469110Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5469455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5469802Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5470174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5470539Z return func(*args, **kwargs) 2025-12-04T09:37:08.5470890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5471356Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5471531Z 2025-12-04T09:37:08.5471632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5471977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5472455Z res = mod(**inputs) 2025-12-04T09:37:08.5472818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5473198Z outputs = self.model.decoder( 2025-12-04T09:37:08.5473563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5473944Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5474293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5474728Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5475099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5475468Z return func(*args, **kwargs) 2025-12-04T09:37:08.5475833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5476254Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5476633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5476979Z return self.act(input) 2025-12-04T09:37:08.5477090Z 2025-12-04T09:37:08.5477213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5477550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5477867Z res = mod(**inputs) 2025-12-04T09:37:08.5478225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5478611Z outputs = self.model.decoder( 2025-12-04T09:37:08.5478975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5479351Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5479696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5480044Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5480419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5480785Z return func(*args, **kwargs) 2025-12-04T09:37:08.5481148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5481528Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5481672Z 2025-12-04T09:37:08.5481777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5482133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5482449Z res = mod(**inputs) 2025-12-04T09:37:08.5482791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5483174Z outputs = self.model.decoder( 2025-12-04T09:37:08.5483555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5483931Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5484283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5484646Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5485069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5485427Z return func(*args, **kwargs) 2025-12-04T09:37:08.5485786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5486186Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5486575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5486939Z return func(*args, **kwargs) 2025-12-04T09:37:08.5487310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5487770Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5488049Z 2025-12-04T09:37:08.5488153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5488515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5488831Z res = mod(**inputs) 2025-12-04T09:37:08.5489189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5489567Z outputs = self.model.decoder( 2025-12-04T09:37:08.5489942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5490322Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5490663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5491043Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5491505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5491936Z return func(*args, **kwargs) 2025-12-04T09:37:08.5492344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5492793Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5493217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5493575Z return func(*args, **kwargs) 2025-12-04T09:37:08.5493932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5494321Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5494463Z 2025-12-04T09:37:08.5494581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5494947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5495308Z res = mod(**inputs) 2025-12-04T09:37:08.5495685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5496084Z outputs = self.model.decoder( 2025-12-04T09:37:08.5496472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5496876Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5497244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5497617Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5498025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5498421Z return func(*args, **kwargs) 2025-12-04T09:37:08.5498809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5499236Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5499704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5500113Z return func(*args, **kwargs) 2025-12-04T09:37:08.5500497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5500898Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5501040Z 2025-12-04T09:37:08.5501115Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5501344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5501680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5502001Z res = mod(**inputs) 2025-12-04T09:37:08.5502337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5502740Z outputs = self.model.decoder( 2025-12-04T09:37:08.5503105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5503477Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5503821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5504156Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5504514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5504864Z return func(*args, **kwargs) 2025-12-04T09:37:08.5505213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5505593Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5505972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5506328Z return func(*args, **kwargs) 2025-12-04T09:37:08.5506669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5507054Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5507489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5507972Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5508143Z 2025-12-04T09:37:08.5508244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5508587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5508896Z res = mod(**inputs) 2025-12-04T09:37:08.5509241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5509604Z outputs = self.model.decoder( 2025-12-04T09:37:08.5509962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5510326Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5510653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5511001Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5511362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5511724Z return func(*args, **kwargs) 2025-12-04T09:37:08.5512076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5512474Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5512899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5513256Z return func(*args, **kwargs) 2025-12-04T09:37:08.5513613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5513995Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5514126Z 2025-12-04T09:37:08.5514231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5514568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5514878Z res = mod(**inputs) 2025-12-04T09:37:08.5515219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5515596Z outputs = self.model.decoder( 2025-12-04T09:37:08.5516005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5516384Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5516729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5517076Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5517448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5517811Z return func(*args, **kwargs) 2025-12-04T09:37:08.5518284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5518700Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5518876Z 2025-12-04T09:37:08.5518979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5519347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5519714Z res = mod(**inputs) 2025-12-04T09:37:08.5520072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5520461Z outputs = self.model.decoder( 2025-12-04T09:37:08.5520839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5521215Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5521578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5521936Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5522313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5522692Z return func(*args, **kwargs) 2025-12-04T09:37:08.5523057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5523489Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5523858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5524199Z return self.act(input) 2025-12-04T09:37:08.5524316Z 2025-12-04T09:37:08.5524419Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5524767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5525082Z res = mod(**inputs) 2025-12-04T09:37:08.5525438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5525819Z outputs = self.model.decoder( 2025-12-04T09:37:08.5526189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5526570Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5526961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5527332Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5527703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5528085Z return func(*args, **kwargs) 2025-12-04T09:37:08.5528449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5528868Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5529012Z 2025-12-04T09:37:08.5529120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5529505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5529886Z res = mod(**inputs) 2025-12-04T09:37:08.5530259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5530668Z outputs = self.model.decoder( 2025-12-04T09:37:08.5531065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5531542Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5531922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5532322Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5532739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5533140Z return func(*args, **kwargs) 2025-12-04T09:37:08.5533541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5533990Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5534394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5534766Z return func(*args, **kwargs) 2025-12-04T09:37:08.5535152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5535644Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5535867Z 2025-12-04T09:37:08.5535983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5536353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5536701Z res = mod(**inputs) 2025-12-04T09:37:08.5537096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5537502Z outputs = self.model.decoder( 2025-12-04T09:37:08.5537904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5538311Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5538684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5539069Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5539487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5539898Z return func(*args, **kwargs) 2025-12-04T09:37:08.5540283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5540722Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5541156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5541604Z return func(*args, **kwargs) 2025-12-04T09:37:08.5541991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5542402Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5542544Z 2025-12-04T09:37:08.5542672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5543021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5543323Z res = mod(**inputs) 2025-12-04T09:37:08.5543669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5544044Z outputs = self.model.decoder( 2025-12-04T09:37:08.5544401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5544806Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5545149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5545502Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5545860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5546222Z return func(*args, **kwargs) 2025-12-04T09:37:08.5546575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5546960Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5547341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5547701Z return func(*args, **kwargs) 2025-12-04T09:37:08.5548061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5548438Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5548579Z 2025-12-04T09:37:08.5548657Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5548892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5549234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5549532Z res = mod(**inputs) 2025-12-04T09:37:08.5549876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5550250Z outputs = self.model.decoder( 2025-12-04T09:37:08.5550609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5550985Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5551327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5551684Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5552043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5552403Z return func(*args, **kwargs) 2025-12-04T09:37:08.5552760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5553147Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5553533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5553895Z return func(*args, **kwargs) 2025-12-04T09:37:08.5554251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5554641Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5555107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5555592Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5555781Z 2025-12-04T09:37:08.5555892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5556253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5556580Z res = mod(**inputs) 2025-12-04T09:37:08.5556943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5557325Z outputs = self.model.decoder( 2025-12-04T09:37:08.5557708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5558132Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5558486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5558842Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5559216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5559584Z return func(*args, **kwargs) 2025-12-04T09:37:08.5559942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5560346Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5560738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5561104Z return func(*args, **kwargs) 2025-12-04T09:37:08.5561467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5561859Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5561993Z 2025-12-04T09:37:08.5562102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5562454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5562763Z res = mod(**inputs) 2025-12-04T09:37:08.5563118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5563498Z outputs = self.model.decoder( 2025-12-04T09:37:08.5563865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5564243Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5564588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5564945Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5565317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5565684Z return func(*args, **kwargs) 2025-12-04T09:37:08.5566047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5566461Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5566636Z 2025-12-04T09:37:08.5566739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5567086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5567404Z res = mod(**inputs) 2025-12-04T09:37:08.5567751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5568138Z outputs = self.model.decoder( 2025-12-04T09:37:08.5568569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5568978Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5569341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5569721Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5570116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5570497Z return func(*args, **kwargs) 2025-12-04T09:37:08.5570883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5571334Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5571893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5572429Z return self.act(input) 2025-12-04T09:37:08.5572573Z 2025-12-04T09:37:08.5572693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5573084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5573426Z res = mod(**inputs) 2025-12-04T09:37:08.5573823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5574233Z outputs = self.model.decoder( 2025-12-04T09:37:08.5574634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5575034Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5575405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5575810Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5576218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5576611Z return func(*args, **kwargs) 2025-12-04T09:37:08.5576996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5577406Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5577556Z 2025-12-04T09:37:08.5577664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5578036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5578381Z res = mod(**inputs) 2025-12-04T09:37:08.5578757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5579153Z outputs = self.model.decoder( 2025-12-04T09:37:08.5579557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5579962Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5580324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5580718Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5581121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5581519Z return func(*args, **kwargs) 2025-12-04T09:37:08.5581903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5582333Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5582759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5583158Z return func(*args, **kwargs) 2025-12-04T09:37:08.5583608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5584093Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5584307Z 2025-12-04T09:37:08.5584424Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5584792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5585131Z res = mod(**inputs) 2025-12-04T09:37:08.5585493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5585867Z outputs = self.model.decoder( 2025-12-04T09:37:08.5586224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5586653Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5587007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5587371Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5587750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5588122Z return func(*args, **kwargs) 2025-12-04T09:37:08.5588500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5588890Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5589279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5589639Z return func(*args, **kwargs) 2025-12-04T09:37:08.5589998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5590377Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5590519Z 2025-12-04T09:37:08.5590620Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5590966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5591274Z res = mod(**inputs) 2025-12-04T09:37:08.5591628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5592014Z outputs = self.model.decoder( 2025-12-04T09:37:08.5592380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5592758Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5593099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5593455Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5593820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5594179Z return func(*args, **kwargs) 2025-12-04T09:37:08.5594537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5594929Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5595313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5595683Z return func(*args, **kwargs) 2025-12-04T09:37:08.5596043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5596421Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5596562Z 2025-12-04T09:37:08.5596640Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5596872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5597250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5597559Z res = mod(**inputs) 2025-12-04T09:37:08.5597908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5598284Z outputs = self.model.decoder( 2025-12-04T09:37:08.5598651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5599019Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5599347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5599690Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5600044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5600427Z return func(*args, **kwargs) 2025-12-04T09:37:08.5600777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5601159Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5601533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5601885Z return func(*args, **kwargs) 2025-12-04T09:37:08.5602232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5602620Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5603100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5603584Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5603759Z 2025-12-04T09:37:08.5603868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5604219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5604523Z res = mod(**inputs) 2025-12-04T09:37:08.5604858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5605223Z outputs = self.model.decoder( 2025-12-04T09:37:08.5605580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5605957Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5606300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5606646Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5607017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5607383Z return func(*args, **kwargs) 2025-12-04T09:37:08.5607747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5608144Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5608542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5608908Z return func(*args, **kwargs) 2025-12-04T09:37:08.5609271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5609652Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5609793Z 2025-12-04T09:37:08.5609897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5610253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5610609Z res = mod(**inputs) 2025-12-04T09:37:08.5610969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5611446Z outputs = self.model.decoder( 2025-12-04T09:37:08.5611858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5612258Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5612629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5613026Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5613435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5613879Z return func(*args, **kwargs) 2025-12-04T09:37:08.5614264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5614690Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5614863Z 2025-12-04T09:37:08.5614967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5615323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5615641Z res = mod(**inputs) 2025-12-04T09:37:08.5615995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5616369Z outputs = self.model.decoder( 2025-12-04T09:37:08.5616743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5617142Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5617504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5617888Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5618293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5618665Z return func(*args, **kwargs) 2025-12-04T09:37:08.5619024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5619450Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5619837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5620180Z return self.act(input) 2025-12-04T09:37:08.5620291Z 2025-12-04T09:37:08.5620392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5620750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5621071Z res = mod(**inputs) 2025-12-04T09:37:08.5621421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5621806Z outputs = self.model.decoder( 2025-12-04T09:37:08.5622203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5622616Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5622960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5623320Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5623693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5624060Z return func(*args, **kwargs) 2025-12-04T09:37:08.5624430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5624869Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5625009Z 2025-12-04T09:37:08.5625120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5625464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5625783Z res = mod(**inputs) 2025-12-04T09:37:08.5626148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5626553Z outputs = self.model.decoder( 2025-12-04T09:37:08.5626961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5627379Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5627754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5628173Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5628551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5628926Z return func(*args, **kwargs) 2025-12-04T09:37:08.5629292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5629695Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5630092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5630458Z return func(*args, **kwargs) 2025-12-04T09:37:08.5630817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5631272Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5631485Z 2025-12-04T09:37:08.5631586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5631941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5632252Z res = mod(**inputs) 2025-12-04T09:37:08.5632606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5632989Z outputs = self.model.decoder( 2025-12-04T09:37:08.5633350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5633728Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5634072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5634430Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5634801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5635171Z return func(*args, **kwargs) 2025-12-04T09:37:08.5635540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5635944Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5636329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5636696Z return func(*args, **kwargs) 2025-12-04T09:37:08.5637060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5637439Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5637580Z 2025-12-04T09:37:08.5637683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5638049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5638361Z res = mod(**inputs) 2025-12-04T09:37:08.5638735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5639116Z outputs = self.model.decoder( 2025-12-04T09:37:08.5639498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5639877Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5640232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5640596Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5640976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5641340Z return func(*args, **kwargs) 2025-12-04T09:37:08.5641751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5642161Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5642561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5642927Z return func(*args, **kwargs) 2025-12-04T09:37:08.5643293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5643686Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5643824Z 2025-12-04T09:37:08.5643903Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5644143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5644494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5644813Z res = mod(**inputs) 2025-12-04T09:37:08.5645170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5645574Z outputs = self.model.decoder( 2025-12-04T09:37:08.5645970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5646363Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5646729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5647122Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5647538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5647955Z return func(*args, **kwargs) 2025-12-04T09:37:08.5648359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5648792Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5649216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5649603Z return func(*args, **kwargs) 2025-12-04T09:37:08.5650000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5650437Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5650913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5651516Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5651734Z 2025-12-04T09:37:08.5651849Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5652239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5652587Z res = mod(**inputs) 2025-12-04T09:37:08.5653022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5653445Z outputs = self.model.decoder( 2025-12-04T09:37:08.5653848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5654265Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5654640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5655031Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5655436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5655841Z return func(*args, **kwargs) 2025-12-04T09:37:08.5656247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5656756Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5657186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5657591Z return func(*args, **kwargs) 2025-12-04T09:37:08.5657992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5658407Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5658563Z 2025-12-04T09:37:08.5658672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5659056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5659412Z res = mod(**inputs) 2025-12-04T09:37:08.5659791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5660217Z outputs = self.model.decoder( 2025-12-04T09:37:08.5660630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5661054Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5661432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5661824Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5662229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5662624Z return func(*args, **kwargs) 2025-12-04T09:37:08.5663027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5663490Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5663681Z 2025-12-04T09:37:08.5663798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5664181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5664527Z res = mod(**inputs) 2025-12-04T09:37:08.5664919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5665319Z outputs = self.model.decoder( 2025-12-04T09:37:08.5665715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5666119Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5666487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5666859Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5667125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5667200Z return func(*args, **kwargs) 2025-12-04T09:37:08.5667502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5667635Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5667862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5667941Z return self.act(input) 2025-12-04T09:37:08.5667945Z 2025-12-04T09:37:08.5668053Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5668259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5668332Z res = mod(**inputs) 2025-12-04T09:37:08.5668593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5668715Z outputs = self.model.decoder( 2025-12-04T09:37:08.5668982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5669055Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5669296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5669377Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5669633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5669710Z return func(*args, **kwargs) 2025-12-04T09:37:08.5669974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5670064Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5670068Z 2025-12-04T09:37:08.5670176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5670391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5670467Z res = mod(**inputs) 2025-12-04T09:37:08.5670731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5670814Z outputs = self.model.decoder( 2025-12-04T09:37:08.5671080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5671152Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5671392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5671474Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5671730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5671813Z return func(*args, **kwargs) 2025-12-04T09:37:08.5672079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5672190Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5672632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5672710Z return func(*args, **kwargs) 2025-12-04T09:37:08.5672982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5673140Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5673145Z 2025-12-04T09:37:08.5673264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5673472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5673544Z res = mod(**inputs) 2025-12-04T09:37:08.5673888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5673969Z outputs = self.model.decoder( 2025-12-04T09:37:08.5674237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5674320Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5674556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5674647Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5674908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5674979Z return func(*args, **kwargs) 2025-12-04T09:37:08.5675249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5675404Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5675663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5675748Z return func(*args, **kwargs) 2025-12-04T09:37:08.5676012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5676103Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5676107Z 2025-12-04T09:37:08.5676218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5676430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5676504Z res = mod(**inputs) 2025-12-04T09:37:08.5676770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5676855Z outputs = self.model.decoder( 2025-12-04T09:37:08.5677128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5677203Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5677449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5677529Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5677785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5677863Z return func(*args, **kwargs) 2025-12-04T09:37:08.5678126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5678236Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5678489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5678562Z return func(*args, **kwargs) 2025-12-04T09:37:08.5678842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5678934Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5678938Z 2025-12-04T09:37:08.5679023Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5679138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5679346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5679417Z res = mod(**inputs) 2025-12-04T09:37:08.5679685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5679759Z outputs = self.model.decoder( 2025-12-04T09:37:08.5680031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5680108Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5680382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5680468Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5680794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5681193Z return func(*args, **kwargs) 2025-12-04T09:37:08.5681474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5681641Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5681910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5682003Z return func(*args, **kwargs) 2025-12-04T09:37:08.5682330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5682480Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5682820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5683008Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5683012Z 2025-12-04T09:37:08.5683143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5683391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5683468Z res = mod(**inputs) 2025-12-04T09:37:08.5683766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5683906Z outputs = self.model.decoder( 2025-12-04T09:37:08.5684184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5684310Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5684557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5684652Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5684982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5685071Z return func(*args, **kwargs) 2025-12-04T09:37:08.5685370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5685492Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5685784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5685922Z return func(*args, **kwargs) 2025-12-04T09:37:08.5686204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5686309Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5686342Z 2025-12-04T09:37:08.5686469Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5686694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5686797Z res = mod(**inputs) 2025-12-04T09:37:08.5687112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5687221Z outputs = self.model.decoder( 2025-12-04T09:37:08.5687552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5687647Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5687927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5688060Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5688371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5688505Z return func(*args, **kwargs) 2025-12-04T09:37:08.5688787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5688936Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5688977Z 2025-12-04T09:37:08.5689111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5689331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5689456Z res = mod(**inputs) 2025-12-04T09:37:08.5707494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5707717Z outputs = self.model.decoder( 2025-12-04T09:37:08.5708075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5708169Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5708422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5708509Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5708765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5708841Z return func(*args, **kwargs) 2025-12-04T09:37:08.5709100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5709229Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5709452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5709535Z return self.act(input) 2025-12-04T09:37:08.5709543Z 2025-12-04T09:37:08.5709657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5709875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5709944Z res = mod(**inputs) 2025-12-04T09:37:08.5710200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5710290Z outputs = self.model.decoder( 2025-12-04T09:37:08.5710537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5710609Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5710849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5710935Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5711196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5711269Z return func(*args, **kwargs) 2025-12-04T09:37:08.5711521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5711614Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5711618Z 2025-12-04T09:37:08.5711726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5711939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5712006Z res = mod(**inputs) 2025-12-04T09:37:08.5712256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5712345Z outputs = self.model.decoder( 2025-12-04T09:37:08.5712785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5712863Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5713100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5713182Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5713441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5713509Z return func(*args, **kwargs) 2025-12-04T09:37:08.5713761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5713879Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5714178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5714252Z return func(*args, **kwargs) 2025-12-04T09:37:08.5714506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5714661Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5714666Z 2025-12-04T09:37:08.5714778Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5714975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5715039Z res = mod(**inputs) 2025-12-04T09:37:08.5715295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5715368Z outputs = self.model.decoder( 2025-12-04T09:37:08.5715624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5715699Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5715924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5716011Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5716248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5716316Z return func(*args, **kwargs) 2025-12-04T09:37:08.5716568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5716668Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5716917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5716987Z return func(*args, **kwargs) 2025-12-04T09:37:08.5717232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5717325Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5717329Z 2025-12-04T09:37:08.5717432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5717641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5717706Z res = mod(**inputs) 2025-12-04T09:37:08.5717953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5718032Z outputs = self.model.decoder( 2025-12-04T09:37:08.5718283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5718355Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5718590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5718668Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5718946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5719017Z return func(*args, **kwargs) 2025-12-04T09:37:08.5719260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5719366Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5719605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5719672Z return func(*args, **kwargs) 2025-12-04T09:37:08.5719922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5720042Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5720046Z 2025-12-04T09:37:08.5720135Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5720242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5720441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5720514Z res = mod(**inputs) 2025-12-04T09:37:08.5720766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5720845Z outputs = self.model.decoder( 2025-12-04T09:37:08.5721093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5721162Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5721392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5721475Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5721720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5721795Z return func(*args, **kwargs) 2025-12-04T09:37:08.5722039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5722143Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5722396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5722486Z return func(*args, **kwargs) 2025-12-04T09:37:08.5722762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5722862Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5723168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5723309Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5723314Z 2025-12-04T09:37:08.5723413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5723611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5723674Z res = mod(**inputs) 2025-12-04T09:37:08.5723916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5723995Z outputs = self.model.decoder( 2025-12-04T09:37:08.5724237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5724313Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5724528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5724608Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5724889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5724959Z return func(*args, **kwargs) 2025-12-04T09:37:08.5725199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5725302Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5725539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5725612Z return func(*args, **kwargs) 2025-12-04T09:37:08.5725855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5725936Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5725973Z 2025-12-04T09:37:08.5726087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5726286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5726360Z res = mod(**inputs) 2025-12-04T09:37:08.5726609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5726681Z outputs = self.model.decoder( 2025-12-04T09:37:08.5726936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5727007Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5727232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5727316Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5727555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5727633Z return func(*args, **kwargs) 2025-12-04T09:37:08.5727882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5728003Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5728007Z 2025-12-04T09:37:08.5728117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5728313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5728386Z res = mod(**inputs) 2025-12-04T09:37:08.5728639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5728712Z outputs = self.model.decoder( 2025-12-04T09:37:08.5728977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5729049Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5729278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5729363Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5729609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5729682Z return func(*args, **kwargs) 2025-12-04T09:37:08.5729937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5730055Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5730284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5730354Z return self.act(input) 2025-12-04T09:37:08.5730358Z 2025-12-04T09:37:08.5730470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5730674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5730767Z res = mod(**inputs) 2025-12-04T09:37:08.5731026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5731098Z outputs = self.model.decoder( 2025-12-04T09:37:08.5731371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5731545Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5731788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5731881Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5732144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5732259Z return func(*args, **kwargs) 2025-12-04T09:37:08.5732541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5732628Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5732633Z 2025-12-04T09:37:08.5732745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5732970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5733038Z res = mod(**inputs) 2025-12-04T09:37:08.5733315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5733402Z outputs = self.model.decoder( 2025-12-04T09:37:08.5733666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5733758Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5733980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5734060Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5734309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5734375Z return func(*args, **kwargs) 2025-12-04T09:37:08.5734621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5734725Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5734963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5735038Z return func(*args, **kwargs) 2025-12-04T09:37:08.5735284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5735437Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5735441Z 2025-12-04T09:37:08.5735553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5735749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5735818Z res = mod(**inputs) 2025-12-04T09:37:08.5736067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5736135Z outputs = self.model.decoder( 2025-12-04T09:37:08.5736391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5736461Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5736681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5736773Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5737057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5737135Z return func(*args, **kwargs) 2025-12-04T09:37:08.5737385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5737481Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5737733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5737799Z return func(*args, **kwargs) 2025-12-04T09:37:08.5738053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5738133Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5738136Z 2025-12-04T09:37:08.5738277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5738478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5738545Z res = mod(**inputs) 2025-12-04T09:37:08.5738793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5738873Z outputs = self.model.decoder( 2025-12-04T09:37:08.5739119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5739194Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5739415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5739492Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5739737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5739807Z return func(*args, **kwargs) 2025-12-04T09:37:08.5740055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5740159Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5740397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5740470Z return func(*args, **kwargs) 2025-12-04T09:37:08.5740715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5740798Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5740802Z 2025-12-04T09:37:08.5740893Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5740994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5741194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5741262Z res = mod(**inputs) 2025-12-04T09:37:08.5741565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5741644Z outputs = self.model.decoder( 2025-12-04T09:37:08.5741891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5741961Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5742187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5742263Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5742510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5742576Z return func(*args, **kwargs) 2025-12-04T09:37:08.5742823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5742962Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5743210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5743277Z return func(*args, **kwargs) 2025-12-04T09:37:08.5743535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5743633Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5743936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5744073Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5744077Z 2025-12-04T09:37:08.5744263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5744499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5744570Z res = mod(**inputs) 2025-12-04T09:37:08.5744827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5744896Z outputs = self.model.decoder( 2025-12-04T09:37:08.5745150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5745222Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5745453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5745528Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5745761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5745834Z return func(*args, **kwargs) 2025-12-04T09:37:08.5746073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5746174Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5746405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5746467Z return func(*args, **kwargs) 2025-12-04T09:37:08.5746714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5746792Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5746796Z 2025-12-04T09:37:08.5746895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5747091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5747150Z res = mod(**inputs) 2025-12-04T09:37:08.5747398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5747469Z outputs = self.model.decoder( 2025-12-04T09:37:08.5747708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5747781Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5747994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5748073Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5748309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5748373Z return func(*args, **kwargs) 2025-12-04T09:37:08.5748618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5748734Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5748737Z 2025-12-04T09:37:08.5748836Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5749065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5749128Z res = mod(**inputs) 2025-12-04T09:37:08.5749383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5749452Z outputs = self.model.decoder( 2025-12-04T09:37:08.5749692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5749767Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5749977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5750051Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5750322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5750391Z return func(*args, **kwargs) 2025-12-04T09:37:08.5750637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5750746Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5750953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5751026Z return self.act(input) 2025-12-04T09:37:08.5751030Z 2025-12-04T09:37:08.5751128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5751323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5751384Z res = mod(**inputs) 2025-12-04T09:37:08.5751624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5751703Z outputs = self.model.decoder( 2025-12-04T09:37:08.5751944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5752013Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5752231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5752307Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5752545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5752609Z return func(*args, **kwargs) 2025-12-04T09:37:08.5752852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5752937Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5752943Z 2025-12-04T09:37:08.5753041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5753245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5753305Z res = mod(**inputs) 2025-12-04T09:37:08.5753549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5753626Z outputs = self.model.decoder( 2025-12-04T09:37:08.5753871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5753938Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5754164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5754240Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5754482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5754552Z return func(*args, **kwargs) 2025-12-04T09:37:08.5754858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5754966Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5755209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5755277Z return func(*args, **kwargs) 2025-12-04T09:37:08.5755532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T09:37:08.5755686Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:37:08.5755689Z 2025-12-04T09:37:08.5755798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5755991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5756099Z res = mod(**inputs) 2025-12-04T09:37:08.5756357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5756428Z outputs = self.model.decoder( 2025-12-04T09:37:08.5756680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5756750Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5756966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5757052Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5757290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5757357Z return func(*args, **kwargs) 2025-12-04T09:37:08.5757615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5757714Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5757959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5758026Z return func(*args, **kwargs) 2025-12-04T09:37:08.5758269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T09:37:08.5758357Z key_states = self.k_proj(current_states) 2025-12-04T09:37:08.5758360Z 2025-12-04T09:37:08.5758457Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5758655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5758717Z res = mod(**inputs) 2025-12-04T09:37:08.5758970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5759046Z outputs = self.model.decoder( 2025-12-04T09:37:08.5759285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5759352Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5759568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5759642Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5759877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5759940Z return func(*args, **kwargs) 2025-12-04T09:37:08.5760177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5760275Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5760506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5760600Z return func(*args, **kwargs) 2025-12-04T09:37:08.5760850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T09:37:08.5760932Z value_states = self.v_proj(current_states) 2025-12-04T09:37:08.5760936Z 2025-12-04T09:37:08.5761020Z cudagraph partition due to non gpu ops 2025-12-04T09:37:08.5761114Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5761300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5761366Z res = mod(**inputs) 2025-12-04T09:37:08.5761607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5761681Z outputs = self.model.decoder( 2025-12-04T09:37:08.5761952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5762023Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5762243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5762318Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5762550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5762624Z return func(*args, **kwargs) 2025-12-04T09:37:08.5762862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5762960Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5763193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5763259Z return func(*args, **kwargs) 2025-12-04T09:37:08.5763505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T09:37:08.5763601Z attn_output, attn_weights = attention_interface( 2025-12-04T09:37:08.5763901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:37:08.5764033Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:08.5764037Z 2025-12-04T09:37:08.5764136Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5764335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5764397Z res = mod(**inputs) 2025-12-04T09:37:08.5764649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5764731Z outputs = self.model.decoder( 2025-12-04T09:37:08.5764983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5765061Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5765278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5765353Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5765596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5765660Z return func(*args, **kwargs) 2025-12-04T09:37:08.5765903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T09:37:08.5766009Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:37:08.5766248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5766325Z return func(*args, **kwargs) 2025-12-04T09:37:08.5766602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T09:37:08.5766686Z attn_output = self.out_proj(attn_output) 2025-12-04T09:37:08.5766689Z 2025-12-04T09:37:08.5766801Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5766995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5767064Z res = mod(**inputs) 2025-12-04T09:37:08.5767330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5767408Z outputs = self.model.decoder( 2025-12-04T09:37:08.5767682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5767793Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5768028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5768116Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5768382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5768457Z return func(*args, **kwargs) 2025-12-04T09:37:08.5768719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5768841Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5768844Z 2025-12-04T09:37:08.5768959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5769166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5769240Z res = mod(**inputs) 2025-12-04T09:37:08.5769518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5769592Z outputs = self.model.decoder( 2025-12-04T09:37:08.5769870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5769942Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5770174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5770261Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5770524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5770601Z return func(*args, **kwargs) 2025-12-04T09:37:08.5771009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T09:37:08.5771144Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:37:08.5771432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:08.5771517Z return self.act(input) 2025-12-04T09:37:08.5771521Z 2025-12-04T09:37:08.5771638Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5771850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5771916Z res = mod(**inputs) 2025-12-04T09:37:08.5772199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T09:37:08.5772474Z outputs = self.model.decoder( 2025-12-04T09:37:08.5772783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T09:37:08.5772872Z layer_outputs = decoder_layer( 2025-12-04T09:37:08.5773199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:08.5773294Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:08.5773552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:08.5773627Z return func(*args, **kwargs) 2025-12-04T09:37:08.5773897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T09:37:08.5773987Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:37:08.5773991Z 2025-12-04T09:37:08.5774103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5774321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5774391Z res = mod(**inputs) 2025-12-04T09:37:08.5774714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1918, in forward 2025-12-04T09:37:08.5774794Z logits = self.lm_head(outputs[0]) 2025-12-04T09:37:08.5774798Z 2025-12-04T09:37:08.5774900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:08.5775099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:08.5775161Z res = mod(**inputs) 2025-12-04T09:37:08.5775413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1924, in forward 2025-12-04T09:37:08.5775559Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:37:08.5775562Z 2025-12-04T09:37:18.9653549Z Compilation time (from dynamo_timed): 16.054110276 2025-12-04T09:37:18.9916998Z pass 2025-12-04T09:37:18.9917577Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:37:18.9919211Z TIMING: _recursive_pre_grad_passes:0.00658 _recursive_joint_graph_passes:0.66444 _recursive_post_grad_passes:0.06407 async_compile.wait:0.80808 code_gen:9.12421 inductor_compile:10.36045 backend_compile:13.40819 gc:0.00016 entire_frame_compile:16.05411 total_wall_time:16.05411 2025-12-04T09:37:18.9920176Z STATS: call_* op count: 336 | FakeTensorMode.__torch_dispatch__:7356 | FakeTensor.__torch_dispatch__:4394 | ProxyTorchDispatchMode.__torch_dispatch__:2034 2025-12-04T09:37:18.9920654Z Dynamo produced 1 graphs covering 336 ops with 0 graph breaks (0 unique) 2025-12-04T09:37:21.4088357Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:37:21.4089242Z import pynvml # type: ignore[import] 2025-12-04T09:37:24.7501279Z 2025-12-04T09:37:25.7611686Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:37:25.7611965Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:37:25.7627843Z cpu eval BertForMaskedLM 2025-12-04T09:37:26.2727176Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:37:26.5095961Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:37:26.7482005Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:37:33.8661262Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8661806Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8662067Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8662405Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8663059Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8663857Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8664221Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8664425Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8664670Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8665317Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8665566Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8665772Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8666044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8666468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8666840Z res = mod(**inputs) 2025-12-04T09:37:33.8667308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8667740Z outputs = self.bert( 2025-12-04T09:37:33.8668152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8668839Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8669275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8669693Z layer_outputs = layer_module( 2025-12-04T09:37:33.8670045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8670423Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8670837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8671242Z return func(*args, **kwargs) 2025-12-04T09:37:33.8671640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8672073Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8672809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8673252Z return func(*args, **kwargs) 2025-12-04T09:37:33.8673633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8674019Z self_outputs = self.self( 2025-12-04T09:37:33.8674384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8674751Z return func(*args, **kwargs) 2025-12-04T09:37:33.8675141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.8675743Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.8676031Z 2025-12-04T09:37:33.8676154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8676539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8676895Z res = mod(**inputs) 2025-12-04T09:37:33.8677279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8677695Z outputs = self.bert( 2025-12-04T09:37:33.8678096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8678510Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8678913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8679331Z layer_outputs = layer_module( 2025-12-04T09:37:33.8679708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8680107Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8680511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8680913Z return func(*args, **kwargs) 2025-12-04T09:37:33.8681392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8681829Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8682242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8682661Z return func(*args, **kwargs) 2025-12-04T09:37:33.8683078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8683616Z self_outputs = self.self( 2025-12-04T09:37:33.8684024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8684421Z return func(*args, **kwargs) 2025-12-04T09:37:33.8684879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.8685294Z self.key(current_states) 2025-12-04T09:37:33.8685419Z 2025-12-04T09:37:33.8685531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8685923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8686286Z res = mod(**inputs) 2025-12-04T09:37:33.8686667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8687074Z outputs = self.bert( 2025-12-04T09:37:33.8687464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8687897Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8688317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8688753Z layer_outputs = layer_module( 2025-12-04T09:37:33.8689134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8689525Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8689935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8690345Z return func(*args, **kwargs) 2025-12-04T09:37:33.8690741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8691173Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8691681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8692098Z return func(*args, **kwargs) 2025-12-04T09:37:33.8692507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8692979Z self_outputs = self.self( 2025-12-04T09:37:33.8693358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8693761Z return func(*args, **kwargs) 2025-12-04T09:37:33.8694143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.8694555Z self.value(current_states) 2025-12-04T09:37:33.8694685Z 2025-12-04T09:37:33.8694770Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8695023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8695398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8695738Z res = mod(**inputs) 2025-12-04T09:37:33.8696118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8696523Z outputs = self.bert( 2025-12-04T09:37:33.8696945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8697367Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8697767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8698175Z layer_outputs = layer_module( 2025-12-04T09:37:33.8698540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8698937Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8699333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8699760Z return func(*args, **kwargs) 2025-12-04T09:37:33.8700143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8700553Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8700950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8701339Z return func(*args, **kwargs) 2025-12-04T09:37:33.8701726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8702127Z self_outputs = self.self( 2025-12-04T09:37:33.8702497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8702885Z return func(*args, **kwargs) 2025-12-04T09:37:33.8703271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.8703731Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.8703950Z 2025-12-04T09:37:33.8704063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8704440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8704780Z res = mod(**inputs) 2025-12-04T09:37:33.8705153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8705550Z outputs = self.bert( 2025-12-04T09:37:33.8706198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8706613Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8706978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8707361Z layer_outputs = layer_module( 2025-12-04T09:37:33.8707711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8708069Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8708446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8708812Z return func(*args, **kwargs) 2025-12-04T09:37:33.8709178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8709558Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8709952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8710343Z return func(*args, **kwargs) 2025-12-04T09:37:33.8710720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.8711185Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.8711715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.8712140Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8712286Z 2025-12-04T09:37:33.8712398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8712777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8713125Z res = mod(**inputs) 2025-12-04T09:37:33.8713518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8713894Z outputs = self.bert( 2025-12-04T09:37:33.8714259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8714710Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8715084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8715469Z layer_outputs = layer_module( 2025-12-04T09:37:33.8715825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8716190Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8716564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8716936Z return func(*args, **kwargs) 2025-12-04T09:37:33.8717309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8717703Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8718105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8718508Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8718924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8719413Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8719879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.8720306Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8720443Z 2025-12-04T09:37:33.8720554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8720908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8721232Z res = mod(**inputs) 2025-12-04T09:37:33.8721589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8721966Z outputs = self.bert( 2025-12-04T09:37:33.8722321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8722704Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8723085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8723462Z layer_outputs = layer_module( 2025-12-04T09:37:33.8723811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8724176Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8724558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8724924Z return func(*args, **kwargs) 2025-12-04T09:37:33.8725299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8725725Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8726122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8726517Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8726925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8727380Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8727800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.8728219Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.8728605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.8728986Z return self.act(input) 2025-12-04T09:37:33.8729099Z 2025-12-04T09:37:33.8729205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8729565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8729887Z res = mod(**inputs) 2025-12-04T09:37:33.8730235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8730613Z outputs = self.bert( 2025-12-04T09:37:33.8730968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8731534Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8731959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8732390Z layer_outputs = layer_module( 2025-12-04T09:37:33.8732781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8733185Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8733595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8733995Z return func(*args, **kwargs) 2025-12-04T09:37:33.8734394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8734817Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8735257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8735688Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8736134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.8736643Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.8737120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.8737546Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8737694Z 2025-12-04T09:37:33.8737805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8738192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8738540Z res = mod(**inputs) 2025-12-04T09:37:33.8738927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8739329Z outputs = self.bert( 2025-12-04T09:37:33.8739714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8740136Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8740576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8740985Z layer_outputs = layer_module( 2025-12-04T09:37:33.8741357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8741736Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8742128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8742521Z return func(*args, **kwargs) 2025-12-04T09:37:33.8742905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8743318Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8743711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8744132Z return func(*args, **kwargs) 2025-12-04T09:37:33.8744521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8744913Z self_outputs = self.self( 2025-12-04T09:37:33.8745296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8745682Z return func(*args, **kwargs) 2025-12-04T09:37:33.8746066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.8746610Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.8746899Z 2025-12-04T09:37:33.8747010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8747394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8747714Z res = mod(**inputs) 2025-12-04T09:37:33.8748068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8748446Z outputs = self.bert( 2025-12-04T09:37:33.8748803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8749181Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8749557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8749933Z layer_outputs = layer_module( 2025-12-04T09:37:33.8750284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8750635Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8751016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8751386Z return func(*args, **kwargs) 2025-12-04T09:37:33.8751745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8752137Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8752515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8752881Z return func(*args, **kwargs) 2025-12-04T09:37:33.8753240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8753622Z self_outputs = self.self( 2025-12-04T09:37:33.8753983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8754346Z return func(*args, **kwargs) 2025-12-04T09:37:33.8754768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.8755173Z self.key(current_states) 2025-12-04T09:37:33.8755287Z 2025-12-04T09:37:33.8755399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8755754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8756073Z res = mod(**inputs) 2025-12-04T09:37:33.8756428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8756804Z outputs = self.bert( 2025-12-04T09:37:33.8757153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8757710Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8758143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8758517Z layer_outputs = layer_module( 2025-12-04T09:37:33.8758867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8759225Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8759603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8759960Z return func(*args, **kwargs) 2025-12-04T09:37:33.8760321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8760711Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8761080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8761450Z return func(*args, **kwargs) 2025-12-04T09:37:33.8761818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8762199Z self_outputs = self.self( 2025-12-04T09:37:33.8762552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8762920Z return func(*args, **kwargs) 2025-12-04T09:37:33.8763295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.8763662Z self.value(current_states) 2025-12-04T09:37:33.8763782Z 2025-12-04T09:37:33.8763860Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8764093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8764442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8764751Z res = mod(**inputs) 2025-12-04T09:37:33.8765097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8765468Z outputs = self.bert( 2025-12-04T09:37:33.8765814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8766189Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8766558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8766937Z layer_outputs = layer_module( 2025-12-04T09:37:33.8767272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8767637Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8768014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8768386Z return func(*args, **kwargs) 2025-12-04T09:37:33.8768771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8769165Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8769548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8769908Z return func(*args, **kwargs) 2025-12-04T09:37:33.8770277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8770654Z self_outputs = self.self( 2025-12-04T09:37:33.8771018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8771487Z return func(*args, **kwargs) 2025-12-04T09:37:33.8771897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.8772607Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.8772813Z 2025-12-04T09:37:33.8772934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8773317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8773663Z res = mod(**inputs) 2025-12-04T09:37:33.8774031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8774396Z outputs = self.bert( 2025-12-04T09:37:33.8774750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8775127Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8775495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8775865Z layer_outputs = layer_module( 2025-12-04T09:37:33.8776204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8776562Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8776923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8777287Z return func(*args, **kwargs) 2025-12-04T09:37:33.8777644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8778028Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8778397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8778766Z return func(*args, **kwargs) 2025-12-04T09:37:33.8779124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.8779554Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.8779982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.8780370Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8780505Z 2025-12-04T09:37:33.8780612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8780954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8781269Z res = mod(**inputs) 2025-12-04T09:37:33.8781616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8781985Z outputs = self.bert( 2025-12-04T09:37:33.8782336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8782705Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8783139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8783504Z layer_outputs = layer_module( 2025-12-04T09:37:33.8783836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8784185Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8784552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8784915Z return func(*args, **kwargs) 2025-12-04T09:37:33.8785282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8785673Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8786105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8786496Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8786900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8787350Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8787758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.8788142Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8788284Z 2025-12-04T09:37:33.8788387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8788736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8789043Z res = mod(**inputs) 2025-12-04T09:37:33.8789393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8789759Z outputs = self.bert( 2025-12-04T09:37:33.8790113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8790477Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8790831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8791191Z layer_outputs = layer_module( 2025-12-04T09:37:33.8791511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8791852Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8792210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8792571Z return func(*args, **kwargs) 2025-12-04T09:37:33.8792924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8793313Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8793691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8794061Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8794447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8794877Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8795286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.8795681Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.8796043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.8796371Z return self.act(input) 2025-12-04T09:37:33.8796475Z 2025-12-04T09:37:33.8796602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8796959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8797282Z res = mod(**inputs) 2025-12-04T09:37:33.8797623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8797978Z outputs = self.bert( 2025-12-04T09:37:33.8798323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8798692Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8799054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8799449Z layer_outputs = layer_module( 2025-12-04T09:37:33.8799784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8800133Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8800487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8800839Z return func(*args, **kwargs) 2025-12-04T09:37:33.8801191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8801566Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8801944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8802316Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8802705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.8803156Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.8803569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.8803947Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8804078Z 2025-12-04T09:37:33.8804186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8804531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8804840Z res = mod(**inputs) 2025-12-04T09:37:33.8805178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8805540Z outputs = self.bert( 2025-12-04T09:37:33.8805879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8806256Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8806625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8806989Z layer_outputs = layer_module( 2025-12-04T09:37:33.8807331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8807688Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8808056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8808412Z return func(*args, **kwargs) 2025-12-04T09:37:33.8808771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8809155Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8809532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8809927Z return func(*args, **kwargs) 2025-12-04T09:37:33.8810298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8810673Z self_outputs = self.self( 2025-12-04T09:37:33.8811030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8811487Z return func(*args, **kwargs) 2025-12-04T09:37:33.8811899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.8812475Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.8812757Z 2025-12-04T09:37:33.8812869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8813297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8813620Z res = mod(**inputs) 2025-12-04T09:37:33.8813963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8814333Z outputs = self.bert( 2025-12-04T09:37:33.8814682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8815055Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8815421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8815793Z layer_outputs = layer_module( 2025-12-04T09:37:33.8816133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8816487Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8816854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8817215Z return func(*args, **kwargs) 2025-12-04T09:37:33.8817576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8817951Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8818321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8818682Z return func(*args, **kwargs) 2025-12-04T09:37:33.8819043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8819402Z self_outputs = self.self( 2025-12-04T09:37:33.8819749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8820112Z return func(*args, **kwargs) 2025-12-04T09:37:33.8820461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.8820828Z self.key(current_states) 2025-12-04T09:37:33.8820943Z 2025-12-04T09:37:33.8821045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8821394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8821698Z res = mod(**inputs) 2025-12-04T09:37:33.8822046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8822414Z outputs = self.bert( 2025-12-04T09:37:33.8822789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8823164Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8823541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8823931Z layer_outputs = layer_module( 2025-12-04T09:37:33.8824257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8824612Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8824975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8825334Z return func(*args, **kwargs) 2025-12-04T09:37:33.8825689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8826079Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8826439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8826836Z return func(*args, **kwargs) 2025-12-04T09:37:33.8827213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8827580Z self_outputs = self.self( 2025-12-04T09:37:33.8827931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8828282Z return func(*args, **kwargs) 2025-12-04T09:37:33.8828640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.8828995Z self.value(current_states) 2025-12-04T09:37:33.8829104Z 2025-12-04T09:37:33.8829181Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8829409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8829752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8830059Z res = mod(**inputs) 2025-12-04T09:37:33.8830390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8830747Z outputs = self.bert( 2025-12-04T09:37:33.8831088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8831444Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8831797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8832154Z layer_outputs = layer_module( 2025-12-04T09:37:33.8832481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8832819Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8833177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8833530Z return func(*args, **kwargs) 2025-12-04T09:37:33.8833875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8834251Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8834607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8834954Z return func(*args, **kwargs) 2025-12-04T09:37:33.8835290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8835647Z self_outputs = self.self( 2025-12-04T09:37:33.8835987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8836337Z return func(*args, **kwargs) 2025-12-04T09:37:33.8836678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.8837127Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.8837304Z 2025-12-04T09:37:33.8837416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8837761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8838072Z res = mod(**inputs) 2025-12-04T09:37:33.8838417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8838784Z outputs = self.bert( 2025-12-04T09:37:33.8839125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8839500Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8839869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8840249Z layer_outputs = layer_module( 2025-12-04T09:37:33.8840581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8840924Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8841281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8841623Z return func(*args, **kwargs) 2025-12-04T09:37:33.8841973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8842346Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8842706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8843048Z return func(*args, **kwargs) 2025-12-04T09:37:33.8843397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.8843807Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.8844207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.8844578Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8844714Z 2025-12-04T09:37:33.8844812Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8845152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8845454Z res = mod(**inputs) 2025-12-04T09:37:33.8845791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8846149Z outputs = self.bert( 2025-12-04T09:37:33.8846482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8846846Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8847202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8847568Z layer_outputs = layer_module( 2025-12-04T09:37:33.8847890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8848241Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8848607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8848974Z return func(*args, **kwargs) 2025-12-04T09:37:33.8849327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8849710Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8850108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8850521Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8850924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8851438Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8851873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.8852308Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8852468Z 2025-12-04T09:37:33.8852591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8852973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8853318Z res = mod(**inputs) 2025-12-04T09:37:33.8853743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8854123Z outputs = self.bert( 2025-12-04T09:37:33.8854485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8854852Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8855216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8855581Z layer_outputs = layer_module( 2025-12-04T09:37:33.8855918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8856261Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8856626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8856991Z return func(*args, **kwargs) 2025-12-04T09:37:33.8857340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8857724Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8858109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8858490Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8858878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8859321Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8859727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.8860132Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.8860505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.8860839Z return self.act(input) 2025-12-04T09:37:33.8860947Z 2025-12-04T09:37:33.8861054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8861395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8861705Z res = mod(**inputs) 2025-12-04T09:37:33.8862049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8862414Z outputs = self.bert( 2025-12-04T09:37:33.8862753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8863125Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8863490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8863853Z layer_outputs = layer_module( 2025-12-04T09:37:33.8864233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8864588Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8864960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8865316Z return func(*args, **kwargs) 2025-12-04T09:37:33.8865678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8866064Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8866454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8866843Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8867264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.8867713Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.8868118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.8868491Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8868626Z 2025-12-04T09:37:33.8868724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8869065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8869386Z res = mod(**inputs) 2025-12-04T09:37:33.8869732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8870098Z outputs = self.bert( 2025-12-04T09:37:33.8870437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8870809Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8871173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8871542Z layer_outputs = layer_module( 2025-12-04T09:37:33.8871874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8872233Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8872753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8873119Z return func(*args, **kwargs) 2025-12-04T09:37:33.8873470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8873851Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8874225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8874584Z return func(*args, **kwargs) 2025-12-04T09:37:33.8874939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8875306Z self_outputs = self.self( 2025-12-04T09:37:33.8875661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8876013Z return func(*args, **kwargs) 2025-12-04T09:37:33.8876379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.8876873Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.8877118Z 2025-12-04T09:37:33.8877229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8877558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8877921Z res = mod(**inputs) 2025-12-04T09:37:33.8878264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8878612Z outputs = self.bert( 2025-12-04T09:37:33.8878952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8879320Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8879688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8880050Z layer_outputs = layer_module( 2025-12-04T09:37:33.8880385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8880796Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8881165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8881526Z return func(*args, **kwargs) 2025-12-04T09:37:33.8881885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8882321Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8882675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8883027Z return func(*args, **kwargs) 2025-12-04T09:37:33.8883385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8883751Z self_outputs = self.self( 2025-12-04T09:37:33.8884111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8884493Z return func(*args, **kwargs) 2025-12-04T09:37:33.8884865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.8885238Z self.key(current_states) 2025-12-04T09:37:33.8885363Z 2025-12-04T09:37:33.8885468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8885829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8886163Z res = mod(**inputs) 2025-12-04T09:37:33.8886514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8886894Z outputs = self.bert( 2025-12-04T09:37:33.8887254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8887635Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8888015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8888396Z layer_outputs = layer_module( 2025-12-04T09:37:33.8888744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8889099Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8889480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8889850Z return func(*args, **kwargs) 2025-12-04T09:37:33.8890210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8890603Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8890984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8891435Z return func(*args, **kwargs) 2025-12-04T09:37:33.8891867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8892277Z self_outputs = self.self( 2025-12-04T09:37:33.8892665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8893037Z return func(*args, **kwargs) 2025-12-04T09:37:33.8893397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.8893779Z self.value(current_states) 2025-12-04T09:37:33.8893896Z 2025-12-04T09:37:33.8893984Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8894220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8894578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8894932Z res = mod(**inputs) 2025-12-04T09:37:33.8895290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8895658Z outputs = self.bert( 2025-12-04T09:37:33.8896014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8896398Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8896767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8897147Z layer_outputs = layer_module( 2025-12-04T09:37:33.8897493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8897857Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8898227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8898602Z return func(*args, **kwargs) 2025-12-04T09:37:33.8898974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8899360Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8899743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8900121Z return func(*args, **kwargs) 2025-12-04T09:37:33.8900478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8900847Z self_outputs = self.self( 2025-12-04T09:37:33.8901204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8901565Z return func(*args, **kwargs) 2025-12-04T09:37:33.8901920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.8902355Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.8902547Z 2025-12-04T09:37:33.8902648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8902998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8903304Z res = mod(**inputs) 2025-12-04T09:37:33.8903654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8904020Z outputs = self.bert( 2025-12-04T09:37:33.8904368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8904735Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8905098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8905473Z layer_outputs = layer_module( 2025-12-04T09:37:33.8905831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8906188Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8906561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8906923Z return func(*args, **kwargs) 2025-12-04T09:37:33.8907275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8907657Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8908027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8908411Z return func(*args, **kwargs) 2025-12-04T09:37:33.8908767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.8909192Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.8909618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.8909995Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8910143Z 2025-12-04T09:37:33.8910244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8910592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8910905Z res = mod(**inputs) 2025-12-04T09:37:33.8911240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8911602Z outputs = self.bert( 2025-12-04T09:37:33.8911950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8912312Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8912674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8913042Z layer_outputs = layer_module( 2025-12-04T09:37:33.8913374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8913719Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8914085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8914443Z return func(*args, **kwargs) 2025-12-04T09:37:33.8914790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8915179Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8915570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8915951Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8916339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8916785Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8917199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.8917588Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8917722Z 2025-12-04T09:37:33.8917833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8918173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8918482Z res = mod(**inputs) 2025-12-04T09:37:33.8918854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8919219Z outputs = self.bert( 2025-12-04T09:37:33.8919555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8919918Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8920263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8920620Z layer_outputs = layer_module( 2025-12-04T09:37:33.8920949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8921288Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8921639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8922028Z return func(*args, **kwargs) 2025-12-04T09:37:33.8922379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8922742Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8923126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8923504Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8923890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8924319Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8924724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.8925122Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.8925483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.8925816Z return self.act(input) 2025-12-04T09:37:33.8925925Z 2025-12-04T09:37:33.8926025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8926363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8926661Z res = mod(**inputs) 2025-12-04T09:37:33.8926997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8927352Z outputs = self.bert( 2025-12-04T09:37:33.8927689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8928053Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8928419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8928794Z layer_outputs = layer_module( 2025-12-04T09:37:33.8929123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8929476Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8929852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8930207Z return func(*args, **kwargs) 2025-12-04T09:37:33.8930549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8930932Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8931313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8931751Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8932176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.8932721Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.8933236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.8933658Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8933817Z 2025-12-04T09:37:33.8933929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8934320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8934672Z res = mod(**inputs) 2025-12-04T09:37:33.8935048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8935458Z outputs = self.bert( 2025-12-04T09:37:33.8935883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8936292Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8936699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8937113Z layer_outputs = layer_module( 2025-12-04T09:37:33.8937483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8937861Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8938262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8938663Z return func(*args, **kwargs) 2025-12-04T09:37:33.8939047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8939471Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8939859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8940210Z return func(*args, **kwargs) 2025-12-04T09:37:33.8940552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8940914Z self_outputs = self.self( 2025-12-04T09:37:33.8941258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8941616Z return func(*args, **kwargs) 2025-12-04T09:37:33.8941962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.8942458Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.8942711Z 2025-12-04T09:37:33.8942819Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8943159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8943468Z res = mod(**inputs) 2025-12-04T09:37:33.8943805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8944164Z outputs = self.bert( 2025-12-04T09:37:33.8944502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8944870Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8945226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8945587Z layer_outputs = layer_module( 2025-12-04T09:37:33.8945909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8946260Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8946653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8947004Z return func(*args, **kwargs) 2025-12-04T09:37:33.8947349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8947714Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8948071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8948413Z return func(*args, **kwargs) 2025-12-04T09:37:33.8948763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8949120Z self_outputs = self.self( 2025-12-04T09:37:33.8949486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8949835Z return func(*args, **kwargs) 2025-12-04T09:37:33.8950181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.8950540Z self.key(current_states) 2025-12-04T09:37:33.8950647Z 2025-12-04T09:37:33.8950745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8951083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8951388Z res = mod(**inputs) 2025-12-04T09:37:33.8951719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8952080Z outputs = self.bert( 2025-12-04T09:37:33.8952416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8952784Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8953134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8953492Z layer_outputs = layer_module( 2025-12-04T09:37:33.8953819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8954162Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8954517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8954866Z return func(*args, **kwargs) 2025-12-04T09:37:33.8955213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8955572Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8955931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8956284Z return func(*args, **kwargs) 2025-12-04T09:37:33.8956632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8956981Z self_outputs = self.self( 2025-12-04T09:37:33.8957321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8957679Z return func(*args, **kwargs) 2025-12-04T09:37:33.8958040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.8958401Z self.value(current_states) 2025-12-04T09:37:33.8958519Z 2025-12-04T09:37:33.8958596Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.8958820Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8959157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8959463Z res = mod(**inputs) 2025-12-04T09:37:33.8959837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8960193Z outputs = self.bert( 2025-12-04T09:37:33.8960531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8960894Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8961257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8961625Z layer_outputs = layer_module( 2025-12-04T09:37:33.8961953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8962296Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8962719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8963077Z return func(*args, **kwargs) 2025-12-04T09:37:33.8963440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8963807Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8964166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8964526Z return func(*args, **kwargs) 2025-12-04T09:37:33.8964881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.8965248Z self_outputs = self.self( 2025-12-04T09:37:33.8965590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8965955Z return func(*args, **kwargs) 2025-12-04T09:37:33.8966313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.8966729Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.8966917Z 2025-12-04T09:37:33.8967016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8967363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8967676Z res = mod(**inputs) 2025-12-04T09:37:33.8968016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8968382Z outputs = self.bert( 2025-12-04T09:37:33.8968731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8969103Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8969466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8969838Z layer_outputs = layer_module( 2025-12-04T09:37:33.8970177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8970521Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8970899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8971260Z return func(*args, **kwargs) 2025-12-04T09:37:33.8971710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.8972087Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.8972646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8973080Z return func(*args, **kwargs) 2025-12-04T09:37:33.8973545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.8974019Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.8974476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.8974864Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8974997Z 2025-12-04T09:37:33.8975099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8975447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8975769Z res = mod(**inputs) 2025-12-04T09:37:33.8976120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8976525Z outputs = self.bert( 2025-12-04T09:37:33.8976875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8977251Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8977607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8977977Z layer_outputs = layer_module( 2025-12-04T09:37:33.8978316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8978667Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8979026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8979385Z return func(*args, **kwargs) 2025-12-04T09:37:33.8979744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8980123Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8980521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8980905Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8981309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8981737Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8982141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.8982516Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8982643Z 2025-12-04T09:37:33.8982748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8983080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8983384Z res = mod(**inputs) 2025-12-04T09:37:33.8983723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8984075Z outputs = self.bert( 2025-12-04T09:37:33.8984415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8984778Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8985133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8985487Z layer_outputs = layer_module( 2025-12-04T09:37:33.8985813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8986158Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8986508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8986866Z return func(*args, **kwargs) 2025-12-04T09:37:33.8987247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8987627Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8988003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8988381Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8988771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.8989209Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.8989608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.8990041Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.8990413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.8990736Z return self.act(input) 2025-12-04T09:37:33.8990849Z 2025-12-04T09:37:33.8990948Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8991290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8991600Z res = mod(**inputs) 2025-12-04T09:37:33.8991934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8992293Z outputs = self.bert( 2025-12-04T09:37:33.8992635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.8992992Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.8993352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.8993715Z layer_outputs = layer_module( 2025-12-04T09:37:33.8994046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.8994391Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.8994751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.8995105Z return func(*args, **kwargs) 2025-12-04T09:37:33.8995455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.8995818Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.8996196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.8996573Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.8996954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.8997401Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.8997819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.8998194Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.8998323Z 2025-12-04T09:37:33.8998421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.8998759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.8999074Z res = mod(**inputs) 2025-12-04T09:37:33.8999413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.8999767Z outputs = self.bert( 2025-12-04T09:37:33.9000109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9000504Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9000864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9001219Z layer_outputs = layer_module( 2025-12-04T09:37:33.9001540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9001875Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9002218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9002559Z return func(*args, **kwargs) 2025-12-04T09:37:33.9002901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9003286Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9003637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9003975Z return func(*args, **kwargs) 2025-12-04T09:37:33.9004311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9004649Z self_outputs = self.self( 2025-12-04T09:37:33.9004981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9005317Z return func(*args, **kwargs) 2025-12-04T09:37:33.9005644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.9006121Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.9006381Z 2025-12-04T09:37:33.9006479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9006828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9007119Z res = mod(**inputs) 2025-12-04T09:37:33.9007452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9007806Z outputs = self.bert( 2025-12-04T09:37:33.9008144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9008497Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9008851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9009210Z layer_outputs = layer_module( 2025-12-04T09:37:33.9009530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9009876Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9010235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9010586Z return func(*args, **kwargs) 2025-12-04T09:37:33.9010923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9011298Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9011792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9012200Z return func(*args, **kwargs) 2025-12-04T09:37:33.9012586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9012997Z self_outputs = self.self( 2025-12-04T09:37:33.9013364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9013751Z return func(*args, **kwargs) 2025-12-04T09:37:33.9014106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.9014468Z self.key(current_states) 2025-12-04T09:37:33.9014578Z 2025-12-04T09:37:33.9014683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9015013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9015315Z res = mod(**inputs) 2025-12-04T09:37:33.9015649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9015998Z outputs = self.bert( 2025-12-04T09:37:33.9016335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9016744Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9017102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9017455Z layer_outputs = layer_module( 2025-12-04T09:37:33.9017787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9018135Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9018487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9018839Z return func(*args, **kwargs) 2025-12-04T09:37:33.9019188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9019562Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9019922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9020280Z return func(*args, **kwargs) 2025-12-04T09:37:33.9020631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9020991Z self_outputs = self.self( 2025-12-04T09:37:33.9021333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9021684Z return func(*args, **kwargs) 2025-12-04T09:37:33.9022029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.9022387Z self.value(current_states) 2025-12-04T09:37:33.9022504Z 2025-12-04T09:37:33.9022581Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.9022807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9023153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9023452Z res = mod(**inputs) 2025-12-04T09:37:33.9023791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9024152Z outputs = self.bert( 2025-12-04T09:37:33.9024484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9024849Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9025206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9025567Z layer_outputs = layer_module( 2025-12-04T09:37:33.9025888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9026231Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9026594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9026976Z return func(*args, **kwargs) 2025-12-04T09:37:33.9027328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9027695Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9028054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9028396Z return func(*args, **kwargs) 2025-12-04T09:37:33.9028743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9029102Z self_outputs = self.self( 2025-12-04T09:37:33.9029432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9029820Z return func(*args, **kwargs) 2025-12-04T09:37:33.9030175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.9030600Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.9030770Z 2025-12-04T09:37:33.9030864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9031198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9031498Z res = mod(**inputs) 2025-12-04T09:37:33.9031833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9032178Z outputs = self.bert( 2025-12-04T09:37:33.9032513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9032877Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9033230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9033589Z layer_outputs = layer_module( 2025-12-04T09:37:33.9033918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9034265Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9034622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9034979Z return func(*args, **kwargs) 2025-12-04T09:37:33.9035336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9035712Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9036073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9036427Z return func(*args, **kwargs) 2025-12-04T09:37:33.9036793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.9037191Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.9037598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.9037966Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9038101Z 2025-12-04T09:37:33.9038210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9038548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9038854Z res = mod(**inputs) 2025-12-04T09:37:33.9039199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9039265Z outputs = self.bert( 2025-12-04T09:37:33.9039538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9039618Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9039857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9039931Z layer_outputs = layer_module( 2025-12-04T09:37:33.9040142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9040217Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9040454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9040518Z return func(*args, **kwargs) 2025-12-04T09:37:33.9040757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9040871Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9041131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9041210Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9041467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9041577Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9041810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.9041886Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9041889Z 2025-12-04T09:37:33.9041990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9042171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9042233Z res = mod(**inputs) 2025-12-04T09:37:33.9042474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9042534Z outputs = self.bert( 2025-12-04T09:37:33.9042774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9042843Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9043074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9043146Z layer_outputs = layer_module( 2025-12-04T09:37:33.9043357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9043431Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9043668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9043733Z return func(*args, **kwargs) 2025-12-04T09:37:33.9043980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9044056Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9044294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9044370Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9044624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9044739Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9044965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.9045072Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.9045313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.9045378Z return self.act(input) 2025-12-04T09:37:33.9045382Z 2025-12-04T09:37:33.9045478Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9045673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9045733Z res = mod(**inputs) 2025-12-04T09:37:33.9045976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9046036Z outputs = self.bert( 2025-12-04T09:37:33.9046274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9046350Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9046611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9046679Z layer_outputs = layer_module( 2025-12-04T09:37:33.9046894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9046967Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9047201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9047268Z return func(*args, **kwargs) 2025-12-04T09:37:33.9047503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9047589Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9047830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9047910Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9048171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.9048295Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.9048536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.9048611Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9048614Z 2025-12-04T09:37:33.9048712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9048907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9048966Z res = mod(**inputs) 2025-12-04T09:37:33.9049206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9049272Z outputs = self.bert( 2025-12-04T09:37:33.9049509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9049587Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9049819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9049891Z layer_outputs = layer_module( 2025-12-04T09:37:33.9050098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9050170Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9050403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9050467Z return func(*args, **kwargs) 2025-12-04T09:37:33.9050695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9050784Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9051043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9051116Z return func(*args, **kwargs) 2025-12-04T09:37:33.9051437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9051513Z self_outputs = self.self( 2025-12-04T09:37:33.9051757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9051823Z return func(*args, **kwargs) 2025-12-04T09:37:33.9052092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.9052325Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.9052368Z 2025-12-04T09:37:33.9052483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9052711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9052777Z res = mod(**inputs) 2025-12-04T09:37:33.9053030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9053106Z outputs = self.bert( 2025-12-04T09:37:33.9053364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9053442Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9053675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9053742Z layer_outputs = layer_module( 2025-12-04T09:37:33.9053963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9054038Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9054266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9054337Z return func(*args, **kwargs) 2025-12-04T09:37:33.9054572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9054655Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9054884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9054947Z return func(*args, **kwargs) 2025-12-04T09:37:33.9055188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9055254Z self_outputs = self.self( 2025-12-04T09:37:33.9055496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9055557Z return func(*args, **kwargs) 2025-12-04T09:37:33.9055792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.9055865Z self.key(current_states) 2025-12-04T09:37:33.9055868Z 2025-12-04T09:37:33.9055963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9056150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9056217Z res = mod(**inputs) 2025-12-04T09:37:33.9056451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9056519Z outputs = self.bert( 2025-12-04T09:37:33.9056752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9056823Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9057097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9057166Z layer_outputs = layer_module( 2025-12-04T09:37:33.9057374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9057455Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9057682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9057751Z return func(*args, **kwargs) 2025-12-04T09:37:33.9057984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9058060Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9058322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9058385Z return func(*args, **kwargs) 2025-12-04T09:37:33.9058621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9058686Z self_outputs = self.self( 2025-12-04T09:37:33.9058911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9058980Z return func(*args, **kwargs) 2025-12-04T09:37:33.9059214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.9059281Z self.value(current_states) 2025-12-04T09:37:33.9059284Z 2025-12-04T09:37:33.9059367Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.9059465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9059657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9059718Z res = mod(**inputs) 2025-12-04T09:37:33.9059950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9060018Z outputs = self.bert( 2025-12-04T09:37:33.9060251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9060317Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9060559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9060626Z layer_outputs = layer_module( 2025-12-04T09:37:33.9060840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9060916Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9061147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9061215Z return func(*args, **kwargs) 2025-12-04T09:37:33.9061450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9061525Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9061757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9061819Z return func(*args, **kwargs) 2025-12-04T09:37:33.9062060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9062125Z self_outputs = self.self( 2025-12-04T09:37:33.9062350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9062423Z return func(*args, **kwargs) 2025-12-04T09:37:33.9062684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.9062820Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.9062824Z 2025-12-04T09:37:33.9062922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9063107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9063173Z res = mod(**inputs) 2025-12-04T09:37:33.9063408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9063470Z outputs = self.bert( 2025-12-04T09:37:33.9063712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9063815Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9064060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9064127Z layer_outputs = layer_module( 2025-12-04T09:37:33.9064338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9064417Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9064641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9064703Z return func(*args, **kwargs) 2025-12-04T09:37:33.9064943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9065019Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9065262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9065325Z return func(*args, **kwargs) 2025-12-04T09:37:33.9065555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.9065682Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.9065912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.9065996Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9066000Z 2025-12-04T09:37:33.9066094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9066278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9066343Z res = mod(**inputs) 2025-12-04T09:37:33.9066578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9066641Z outputs = self.bert( 2025-12-04T09:37:33.9066884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9066952Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9067191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9067255Z layer_outputs = layer_module( 2025-12-04T09:37:33.9067461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9067542Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9067770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9067840Z return func(*args, **kwargs) 2025-12-04T09:37:33.9068074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9068154Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9068443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9068515Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9068769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9068885Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9069112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.9069193Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9069196Z 2025-12-04T09:37:33.9069291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9069505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9069572Z res = mod(**inputs) 2025-12-04T09:37:33.9069809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9069879Z outputs = self.bert( 2025-12-04T09:37:33.9070113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9070180Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9070418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9070485Z layer_outputs = layer_module( 2025-12-04T09:37:33.9070691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9070773Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9071002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9071074Z return func(*args, **kwargs) 2025-12-04T09:37:33.9071305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9071384Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9071635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9071707Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9071966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9072082Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9072429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.9072555Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.9072766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.9072834Z return self.act(input) 2025-12-04T09:37:33.9072838Z 2025-12-04T09:37:33.9072943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9073127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9073193Z res = mod(**inputs) 2025-12-04T09:37:33.9073430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9073491Z outputs = self.bert( 2025-12-04T09:37:33.9073731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9073802Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9074104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9074181Z layer_outputs = layer_module( 2025-12-04T09:37:33.9074390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9074468Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9074695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9074759Z return func(*args, **kwargs) 2025-12-04T09:37:33.9074998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9075073Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9075323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9076382Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9076652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.9076783Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.9077017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.9077092Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9077102Z 2025-12-04T09:37:33.9077201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9077386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9077451Z res = mod(**inputs) 2025-12-04T09:37:33.9077687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9077752Z outputs = self.bert( 2025-12-04T09:37:33.9077995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9078063Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9078305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9078374Z layer_outputs = layer_module( 2025-12-04T09:37:33.9078584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9078664Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9078895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9078960Z return func(*args, **kwargs) 2025-12-04T09:37:33.9079202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9079281Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9079515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9079579Z return func(*args, **kwargs) 2025-12-04T09:37:33.9079811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9079885Z self_outputs = self.self( 2025-12-04T09:37:33.9080113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9080176Z return func(*args, **kwargs) 2025-12-04T09:37:33.9080417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.9080619Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.9080622Z 2025-12-04T09:37:33.9080759Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9080947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9081006Z res = mod(**inputs) 2025-12-04T09:37:33.9081302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9081362Z outputs = self.bert( 2025-12-04T09:37:33.9081607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9081676Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9081910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9081982Z layer_outputs = layer_module( 2025-12-04T09:37:33.9082222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9082297Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9082532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9082595Z return func(*args, **kwargs) 2025-12-04T09:37:33.9082835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9082909Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9083136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9083207Z return func(*args, **kwargs) 2025-12-04T09:37:33.9083438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9083512Z self_outputs = self.self( 2025-12-04T09:37:33.9083743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9083807Z return func(*args, **kwargs) 2025-12-04T09:37:33.9084059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.9084124Z self.key(current_states) 2025-12-04T09:37:33.9084128Z 2025-12-04T09:37:33.9084227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9084420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9084479Z res = mod(**inputs) 2025-12-04T09:37:33.9084722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9084783Z outputs = self.bert( 2025-12-04T09:37:33.9085022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9085100Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9085333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9085399Z layer_outputs = layer_module( 2025-12-04T09:37:33.9085618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9085690Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9085924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9085988Z return func(*args, **kwargs) 2025-12-04T09:37:33.9086219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9086304Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9086560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9086625Z return func(*args, **kwargs) 2025-12-04T09:37:33.9086864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9086927Z self_outputs = self.self( 2025-12-04T09:37:33.9087163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9087223Z return func(*args, **kwargs) 2025-12-04T09:37:33.9087458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.9087532Z self.value(current_states) 2025-12-04T09:37:33.9087535Z 2025-12-04T09:37:33.9087610Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.9087748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9087936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9087996Z res = mod(**inputs) 2025-12-04T09:37:33.9088237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9088302Z outputs = self.bert( 2025-12-04T09:37:33.9088541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9088617Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9088854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9088935Z layer_outputs = layer_module( 2025-12-04T09:37:33.9089143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9089216Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9089451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9089517Z return func(*args, **kwargs) 2025-12-04T09:37:33.9089753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9089836Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9090078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9090150Z return func(*args, **kwargs) 2025-12-04T09:37:33.9090387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9090452Z self_outputs = self.self( 2025-12-04T09:37:33.9090686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9090752Z return func(*args, **kwargs) 2025-12-04T09:37:33.9090993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.9091119Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.9091123Z 2025-12-04T09:37:33.9091220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9091477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9091545Z res = mod(**inputs) 2025-12-04T09:37:33.9091787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9091861Z outputs = self.bert( 2025-12-04T09:37:33.9092126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9092215Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9092525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9092598Z layer_outputs = layer_module( 2025-12-04T09:37:33.9092827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9092906Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9093148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9093224Z return func(*args, **kwargs) 2025-12-04T09:37:33.9093470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9093557Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9093838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9093914Z return func(*args, **kwargs) 2025-12-04T09:37:33.9094158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.9094277Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.9094519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.9094597Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9094601Z 2025-12-04T09:37:33.9094698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9094891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9094952Z res = mod(**inputs) 2025-12-04T09:37:33.9095186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9095256Z outputs = self.bert( 2025-12-04T09:37:33.9095493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9095567Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9095798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9095862Z layer_outputs = layer_module( 2025-12-04T09:37:33.9096078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9096149Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9096384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9096448Z return func(*args, **kwargs) 2025-12-04T09:37:33.9096680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9096767Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9097012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9097082Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9097349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9097461Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9097701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.9097777Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9097780Z 2025-12-04T09:37:33.9097877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9098072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9098159Z res = mod(**inputs) 2025-12-04T09:37:33.9098404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9098466Z outputs = self.bert( 2025-12-04T09:37:33.9098704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9098779Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9099012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9099077Z layer_outputs = layer_module( 2025-12-04T09:37:33.9099293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9099418Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9099654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9099716Z return func(*args, **kwargs) 2025-12-04T09:37:33.9099946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9100033Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9100277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9100347Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9100612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9100724Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9100963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.9101073Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.9101273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.9101346Z return self.act(input) 2025-12-04T09:37:33.9101350Z 2025-12-04T09:37:33.9101445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9101634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9101693Z res = mod(**inputs) 2025-12-04T09:37:33.9101927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9101992Z outputs = self.bert( 2025-12-04T09:37:33.9102225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9102297Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9102539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9102604Z layer_outputs = layer_module( 2025-12-04T09:37:33.9102819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9102891Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9103116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9103187Z return func(*args, **kwargs) 2025-12-04T09:37:33.9103416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9103500Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9103741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9103813Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9104108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.9104234Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.9104466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.9104549Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9104552Z 2025-12-04T09:37:33.9104650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9104839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9104899Z res = mod(**inputs) 2025-12-04T09:37:33.9105137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9105239Z outputs = self.bert( 2025-12-04T09:37:33.9105481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9105556Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9105790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9105857Z layer_outputs = layer_module( 2025-12-04T09:37:33.9106082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9106154Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9106382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9106454Z return func(*args, **kwargs) 2025-12-04T09:37:33.9106697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9106783Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9107012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9107076Z return func(*args, **kwargs) 2025-12-04T09:37:33.9107319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9107387Z self_outputs = self.self( 2025-12-04T09:37:33.9107613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9107683Z return func(*args, **kwargs) 2025-12-04T09:37:33.9107916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.9108124Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.9108127Z 2025-12-04T09:37:33.9108228Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9108414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9108481Z res = mod(**inputs) 2025-12-04T09:37:33.9108718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9108784Z outputs = self.bert( 2025-12-04T09:37:33.9109020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9109088Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9109330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9109401Z layer_outputs = layer_module( 2025-12-04T09:37:33.9109637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9109720Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9109949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9110018Z return func(*args, **kwargs) 2025-12-04T09:37:33.9110255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9110330Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9110566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9110627Z return func(*args, **kwargs) 2025-12-04T09:37:33.9110860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9110961Z self_outputs = self.self( 2025-12-04T09:37:33.9111192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9111263Z return func(*args, **kwargs) 2025-12-04T09:37:33.9111495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.9111560Z self.key(current_states) 2025-12-04T09:37:33.9111564Z 2025-12-04T09:37:33.9111668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9111854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9111919Z res = mod(**inputs) 2025-12-04T09:37:33.9112157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9112221Z outputs = self.bert( 2025-12-04T09:37:33.9112464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9112546Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9112774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9112848Z layer_outputs = layer_module( 2025-12-04T09:37:33.9113055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9113133Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9113354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9113417Z return func(*args, **kwargs) 2025-12-04T09:37:33.9113652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9113729Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9113951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9114022Z return func(*args, **kwargs) 2025-12-04T09:37:33.9114249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9114319Z self_outputs = self.self( 2025-12-04T09:37:33.9114564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9114624Z return func(*args, **kwargs) 2025-12-04T09:37:33.9114857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.9114923Z self.value(current_states) 2025-12-04T09:37:33.9114926Z 2025-12-04T09:37:33.9115011Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.9115108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9115321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9115387Z res = mod(**inputs) 2025-12-04T09:37:33.9115616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9115674Z outputs = self.bert( 2025-12-04T09:37:33.9115906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9115973Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9116205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9116267Z layer_outputs = layer_module( 2025-12-04T09:37:33.9116468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9116588Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9116814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9116876Z return func(*args, **kwargs) 2025-12-04T09:37:33.9117110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9117184Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9117413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9117474Z return func(*args, **kwargs) 2025-12-04T09:37:33.9117708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9117780Z self_outputs = self.self( 2025-12-04T09:37:33.9118009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9118081Z return func(*args, **kwargs) 2025-12-04T09:37:33.9118317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.9118443Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.9118447Z 2025-12-04T09:37:33.9118550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9118735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9118793Z res = mod(**inputs) 2025-12-04T09:37:33.9119037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9119096Z outputs = self.bert( 2025-12-04T09:37:33.9119338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9119407Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9119641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9119715Z layer_outputs = layer_module( 2025-12-04T09:37:33.9119922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9119994Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9120227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9120291Z return func(*args, **kwargs) 2025-12-04T09:37:33.9120530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9120604Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9120830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9120930Z return func(*args, **kwargs) 2025-12-04T09:37:33.9121167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.9121296Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.9121536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.9121615Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9121618Z 2025-12-04T09:37:33.9121723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9121911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9121968Z res = mod(**inputs) 2025-12-04T09:37:33.9122239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9122301Z outputs = self.bert( 2025-12-04T09:37:33.9122541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9122609Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9122841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9122915Z layer_outputs = layer_module( 2025-12-04T09:37:33.9123120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9123197Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9123426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9123491Z return func(*args, **kwargs) 2025-12-04T09:37:33.9123731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9123810Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9124053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9124133Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9124390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9124508Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9124741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.9124817Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9124820Z 2025-12-04T09:37:33.9124928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9125119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9125184Z res = mod(**inputs) 2025-12-04T09:37:33.9125419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9125480Z outputs = self.bert( 2025-12-04T09:37:33.9125726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9125794Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9126032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9126109Z layer_outputs = layer_module( 2025-12-04T09:37:33.9126323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9126408Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9126669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9126738Z return func(*args, **kwargs) 2025-12-04T09:37:33.9126983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9127062Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9127313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9127392Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9127658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9127777Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9128053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.9128166Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.9128385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.9128451Z return self.act(input) 2025-12-04T09:37:33.9128455Z 2025-12-04T09:37:33.9128559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9128755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9128815Z res = mod(**inputs) 2025-12-04T09:37:33.9129071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9129133Z outputs = self.bert( 2025-12-04T09:37:33.9129382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9129460Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9129706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9129782Z layer_outputs = layer_module( 2025-12-04T09:37:33.9129999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9130074Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9130319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9130383Z return func(*args, **kwargs) 2025-12-04T09:37:33.9130627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9130714Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9130973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9131055Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9131399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.9131541Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.9131798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.9131877Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9131881Z 2025-12-04T09:37:33.9131992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9132188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9132252Z res = mod(**inputs) 2025-12-04T09:37:33.9132512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9132614Z outputs = self.bert( 2025-12-04T09:37:33.9132864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9132942Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9133187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9133267Z layer_outputs = layer_module( 2025-12-04T09:37:33.9133487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9133566Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9133815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9133919Z return func(*args, **kwargs) 2025-12-04T09:37:33.9134174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9134255Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9134504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9134577Z return func(*args, **kwargs) 2025-12-04T09:37:33.9134815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9134880Z self_outputs = self.self( 2025-12-04T09:37:33.9135120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9135184Z return func(*args, **kwargs) 2025-12-04T09:37:33.9135430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.9135633Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.9135638Z 2025-12-04T09:37:33.9135738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9135938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9135999Z res = mod(**inputs) 2025-12-04T09:37:33.9136248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9136311Z outputs = self.bert( 2025-12-04T09:37:33.9136552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9136629Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9136873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9136942Z layer_outputs = layer_module( 2025-12-04T09:37:33.9137166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9137240Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9137480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9137545Z return func(*args, **kwargs) 2025-12-04T09:37:33.9137784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9137876Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9138107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9138171Z return func(*args, **kwargs) 2025-12-04T09:37:33.9138416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9138484Z self_outputs = self.self( 2025-12-04T09:37:33.9138762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9138829Z return func(*args, **kwargs) 2025-12-04T09:37:33.9139070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.9139145Z self.key(current_states) 2025-12-04T09:37:33.9139148Z 2025-12-04T09:37:33.9139246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9139443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9139503Z res = mod(**inputs) 2025-12-04T09:37:33.9139744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9139846Z outputs = self.bert( 2025-12-04T09:37:33.9140094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9140166Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9140418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9140487Z layer_outputs = layer_module( 2025-12-04T09:37:33.9140710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9140785Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9141019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9141094Z return func(*args, **kwargs) 2025-12-04T09:37:33.9141333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9141416Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9141664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9141730Z return func(*args, **kwargs) 2025-12-04T09:37:33.9141978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9142043Z self_outputs = self.self( 2025-12-04T09:37:33.9142280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9142353Z return func(*args, **kwargs) 2025-12-04T09:37:33.9142596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.9142672Z self.value(current_states) 2025-12-04T09:37:33.9142679Z 2025-12-04T09:37:33.9142759Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.9142989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9143190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9143250Z res = mod(**inputs) 2025-12-04T09:37:33.9143497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9143566Z outputs = self.bert( 2025-12-04T09:37:33.9143811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9143885Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9144177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9144242Z layer_outputs = layer_module( 2025-12-04T09:37:33.9144461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9144535Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9144808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9144873Z return func(*args, **kwargs) 2025-12-04T09:37:33.9145111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9145196Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9145426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9145489Z return func(*args, **kwargs) 2025-12-04T09:37:33.9145733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9145829Z self_outputs = self.self( 2025-12-04T09:37:33.9146069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9146133Z return func(*args, **kwargs) 2025-12-04T09:37:33.9146369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.9146503Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.9146506Z 2025-12-04T09:37:33.9146603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9146794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9146860Z res = mod(**inputs) 2025-12-04T09:37:33.9147099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9147166Z outputs = self.bert( 2025-12-04T09:37:33.9147411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9147481Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9147728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9147794Z layer_outputs = layer_module( 2025-12-04T09:37:33.9148015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9148088Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9148321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9148392Z return func(*args, **kwargs) 2025-12-04T09:37:33.9148629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9148709Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9148949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9149013Z return func(*args, **kwargs) 2025-12-04T09:37:33.9149256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.9149378Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.9149615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.9149703Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9149706Z 2025-12-04T09:37:33.9149804Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9150001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9150065Z res = mod(**inputs) 2025-12-04T09:37:33.9150304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9150404Z outputs = self.bert( 2025-12-04T09:37:33.9150640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9150707Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9150946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9151009Z layer_outputs = layer_module( 2025-12-04T09:37:33.9151224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9151294Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9151519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9151620Z return func(*args, **kwargs) 2025-12-04T09:37:33.9151859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9151939Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9152194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9152264Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9152535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9152646Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9152880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.9152964Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9152970Z 2025-12-04T09:37:33.9153068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9153261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9153320Z res = mod(**inputs) 2025-12-04T09:37:33.9153558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9153626Z outputs = self.bert( 2025-12-04T09:37:33.9153863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9153930Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9154173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9154239Z layer_outputs = layer_module( 2025-12-04T09:37:33.9154489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9154566Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9154796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9154868Z return func(*args, **kwargs) 2025-12-04T09:37:33.9155102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9155190Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9155438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9155510Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9155783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9155893Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9156162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.9156278Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.9156482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.9156559Z return self.act(input) 2025-12-04T09:37:33.9156562Z 2025-12-04T09:37:33.9156655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9156834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9156898Z res = mod(**inputs) 2025-12-04T09:37:33.9157126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9157184Z outputs = self.bert( 2025-12-04T09:37:33.9157417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9157519Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9157754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9157817Z layer_outputs = layer_module( 2025-12-04T09:37:33.9158017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9158095Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9158313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9158381Z return func(*args, **kwargs) 2025-12-04T09:37:33.9158608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9158686Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9158930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9158999Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9159248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.9159373Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.9159600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.9159680Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9159684Z 2025-12-04T09:37:33.9159776Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9159957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9160025Z res = mod(**inputs) 2025-12-04T09:37:33.9160253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9160323Z outputs = self.bert( 2025-12-04T09:37:33.9160550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9160619Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9160852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9160918Z layer_outputs = layer_module( 2025-12-04T09:37:33.9161118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9161197Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9161417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9161490Z return func(*args, **kwargs) 2025-12-04T09:37:33.9161748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9161824Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9162052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9162114Z return func(*args, **kwargs) 2025-12-04T09:37:33.9162341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9162412Z self_outputs = self.self( 2025-12-04T09:37:33.9162633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9162703Z return func(*args, **kwargs) 2025-12-04T09:37:33.9162929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.9163155Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.9163159Z 2025-12-04T09:37:33.9163262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9163443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9163509Z res = mod(**inputs) 2025-12-04T09:37:33.9163740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9163798Z outputs = self.bert( 2025-12-04T09:37:33.9164035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9164102Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9164329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9164403Z layer_outputs = layer_module( 2025-12-04T09:37:33.9164609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9164688Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9164910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9164971Z return func(*args, **kwargs) 2025-12-04T09:37:33.9165208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9165281Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9165510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9165571Z return func(*args, **kwargs) 2025-12-04T09:37:33.9165805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9165880Z self_outputs = self.self( 2025-12-04T09:37:33.9166103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9166163Z return func(*args, **kwargs) 2025-12-04T09:37:33.9166396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.9166459Z self.key(current_states) 2025-12-04T09:37:33.9166462Z 2025-12-04T09:37:33.9166563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9166744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9166802Z res = mod(**inputs) 2025-12-04T09:37:33.9167041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9167103Z outputs = self.bert( 2025-12-04T09:37:33.9167362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9167440Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9167676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9167749Z layer_outputs = layer_module( 2025-12-04T09:37:33.9167958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9168030Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9168267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9168330Z return func(*args, **kwargs) 2025-12-04T09:37:33.9168599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9168677Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9168903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9168973Z return func(*args, **kwargs) 2025-12-04T09:37:33.9169207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9169270Z self_outputs = self.self( 2025-12-04T09:37:33.9169502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9169564Z return func(*args, **kwargs) 2025-12-04T09:37:33.9169801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.9169872Z self.value(current_states) 2025-12-04T09:37:33.9169875Z 2025-12-04T09:37:33.9169951Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.9170058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9170244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9170303Z res = mod(**inputs) 2025-12-04T09:37:33.9170546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9170606Z outputs = self.bert( 2025-12-04T09:37:33.9170847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9170913Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9171172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9171252Z layer_outputs = layer_module( 2025-12-04T09:37:33.9171555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9171654Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9171923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9171996Z return func(*args, **kwargs) 2025-12-04T09:37:33.9172414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9172509Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9172774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9172854Z return func(*args, **kwargs) 2025-12-04T09:37:33.9173126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9173220Z self_outputs = self.self( 2025-12-04T09:37:33.9173512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9173578Z return func(*args, **kwargs) 2025-12-04T09:37:33.9173821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.9173949Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.9173952Z 2025-12-04T09:37:33.9174050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9174243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9174304Z res = mod(**inputs) 2025-12-04T09:37:33.9174583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9174708Z outputs = self.bert( 2025-12-04T09:37:33.9174981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9175068Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9175344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9175426Z layer_outputs = layer_module( 2025-12-04T09:37:33.9175664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9175745Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9176017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9176089Z return func(*args, **kwargs) 2025-12-04T09:37:33.9176373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9176467Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9176735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9176813Z return func(*args, **kwargs) 2025-12-04T09:37:33.9177091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.9177226Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.9177508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.9177594Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9177598Z 2025-12-04T09:37:33.9177717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9177928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9177996Z res = mod(**inputs) 2025-12-04T09:37:33.9178277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9178344Z outputs = self.bert( 2025-12-04T09:37:33.9178613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9178695Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9178973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9179053Z layer_outputs = layer_module( 2025-12-04T09:37:33.9179292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9179370Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9179640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9179713Z return func(*args, **kwargs) 2025-12-04T09:37:33.9180025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9180125Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9180397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9180484Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9180782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9180908Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9181171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.9181295Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9181298Z 2025-12-04T09:37:33.9181401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9181588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9181648Z res = mod(**inputs) 2025-12-04T09:37:33.9181899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9181959Z outputs = self.bert( 2025-12-04T09:37:33.9182191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9182267Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9182498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9182571Z layer_outputs = layer_module( 2025-12-04T09:37:33.9182778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9182852Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9183087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9183151Z return func(*args, **kwargs) 2025-12-04T09:37:33.9183384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9183471Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9183717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9183794Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9184054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9184168Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9184409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.9184516Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.9184727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.9184792Z return self.act(input) 2025-12-04T09:37:33.9184796Z 2025-12-04T09:37:33.9184891Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9185081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9185142Z res = mod(**inputs) 2025-12-04T09:37:33.9185376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9185443Z outputs = self.bert( 2025-12-04T09:37:33.9185679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9185784Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9186017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9186084Z layer_outputs = layer_module( 2025-12-04T09:37:33.9186298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9186372Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9186605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9186668Z return func(*args, **kwargs) 2025-12-04T09:37:33.9186899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9187012Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9187258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9187327Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9187593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.9187715Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.9187954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.9188029Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9188032Z 2025-12-04T09:37:33.9188127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9188316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9188378Z res = mod(**inputs) 2025-12-04T09:37:33.9188620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9188680Z outputs = self.bert( 2025-12-04T09:37:33.9188910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9188983Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9189212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9189277Z layer_outputs = layer_module( 2025-12-04T09:37:33.9189488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9189559Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9189800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9189866Z return func(*args, **kwargs) 2025-12-04T09:37:33.9190097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9190180Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9190406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9190467Z return func(*args, **kwargs) 2025-12-04T09:37:33.9190703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9190767Z self_outputs = self.self( 2025-12-04T09:37:33.9191000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9191064Z return func(*args, **kwargs) 2025-12-04T09:37:33.9191297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T09:37:33.9191554Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:37:33.9191558Z 2025-12-04T09:37:33.9191659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9191854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9191917Z res = mod(**inputs) 2025-12-04T09:37:33.9192155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9192229Z outputs = self.bert( 2025-12-04T09:37:33.9192466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9192537Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9192806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9192874Z layer_outputs = layer_module( 2025-12-04T09:37:33.9193092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9193165Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9193391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9193462Z return func(*args, **kwargs) 2025-12-04T09:37:33.9193694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9193777Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9194001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9194067Z return func(*args, **kwargs) 2025-12-04T09:37:33.9194306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9194370Z self_outputs = self.self( 2025-12-04T09:37:33.9194595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9194664Z return func(*args, **kwargs) 2025-12-04T09:37:33.9194894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T09:37:33.9194966Z self.key(current_states) 2025-12-04T09:37:33.9194969Z 2025-12-04T09:37:33.9195071Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9195254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9195319Z res = mod(**inputs) 2025-12-04T09:37:33.9195551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9195616Z outputs = self.bert( 2025-12-04T09:37:33.9195856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9195924Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9196160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9196224Z layer_outputs = layer_module( 2025-12-04T09:37:33.9196431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9196513Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9196745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9196816Z return func(*args, **kwargs) 2025-12-04T09:37:33.9197055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9197162Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9197406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9197472Z return func(*args, **kwargs) 2025-12-04T09:37:33.9197711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9197786Z self_outputs = self.self( 2025-12-04T09:37:33.9198018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9198090Z return func(*args, **kwargs) 2025-12-04T09:37:33.9198339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T09:37:33.9198436Z self.value(current_states) 2025-12-04T09:37:33.9198439Z 2025-12-04T09:37:33.9198524Z cudagraph partition due to non gpu ops 2025-12-04T09:37:33.9198624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9198810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9198878Z res = mod(**inputs) 2025-12-04T09:37:33.9199109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9199176Z outputs = self.bert( 2025-12-04T09:37:33.9199412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9199480Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9199724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9199793Z layer_outputs = layer_module( 2025-12-04T09:37:33.9200018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9200092Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9200324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9200396Z return func(*args, **kwargs) 2025-12-04T09:37:33.9200632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9200708Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9200947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9201010Z return func(*args, **kwargs) 2025-12-04T09:37:33.9201253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T09:37:33.9201321Z self_outputs = self.self( 2025-12-04T09:37:33.9201552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9201623Z return func(*args, **kwargs) 2025-12-04T09:37:33.9201859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T09:37:33.9201986Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:37:33.9201996Z 2025-12-04T09:37:33.9202093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9202283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9202349Z res = mod(**inputs) 2025-12-04T09:37:33.9202587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9202651Z outputs = self.bert( 2025-12-04T09:37:33.9202925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9202995Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9203243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9203311Z layer_outputs = layer_module( 2025-12-04T09:37:33.9203528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9203609Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9203848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9203913Z return func(*args, **kwargs) 2025-12-04T09:37:33.9204165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T09:37:33.9204282Z self_attention_outputs = self.attention( 2025-12-04T09:37:33.9204525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9204590Z return func(*args, **kwargs) 2025-12-04T09:37:33.9204829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T09:37:33.9204964Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:37:33.9205203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T09:37:33.9205283Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9205293Z 2025-12-04T09:37:33.9205394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9205582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9205656Z res = mod(**inputs) 2025-12-04T09:37:33.9205902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9205965Z outputs = self.bert( 2025-12-04T09:37:33.9206214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9206282Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9206526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9206594Z layer_outputs = layer_module( 2025-12-04T09:37:33.9206805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9206886Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9207118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9207187Z return func(*args, **kwargs) 2025-12-04T09:37:33.9207438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9207521Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9207785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9207860Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9208134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9208261Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9208504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T09:37:33.9208593Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9208597Z 2025-12-04T09:37:33.9208726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9208926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9208996Z res = mod(**inputs) 2025-12-04T09:37:33.9209245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9209308Z outputs = self.bert( 2025-12-04T09:37:33.9209568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9209638Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9209892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9209960Z layer_outputs = layer_module( 2025-12-04T09:37:33.9210211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9210298Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9210542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9210609Z return func(*args, **kwargs) 2025-12-04T09:37:33.9210862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9210944Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9211210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9211284Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9211634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T09:37:33.9211773Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:37:33.9212021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T09:37:33.9212139Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:37:33.9212353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:37:33.9212422Z return self.act(input) 2025-12-04T09:37:33.9212426Z 2025-12-04T09:37:33.9212535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9212733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9212795Z res = mod(**inputs) 2025-12-04T09:37:33.9213050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T09:37:33.9213117Z outputs = self.bert( 2025-12-04T09:37:33.9213374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T09:37:33.9213446Z encoder_outputs = self.encoder( 2025-12-04T09:37:33.9213690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T09:37:33.9213769Z layer_outputs = layer_module( 2025-12-04T09:37:33.9213987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:37:33.9214071Z return super().__call__(*args, **kwargs) 2025-12-04T09:37:33.9214310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:37:33.9214377Z return func(*args, **kwargs) 2025-12-04T09:37:33.9214629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T09:37:33.9214713Z layer_output = apply_chunking_to_forward( 2025-12-04T09:37:33.9215005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:37:33.9215089Z return forward_fn(*input_tensors) 2025-12-04T09:37:33.9215362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T09:37:33.9215499Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:37:33.9215743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T09:37:33.9215823Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9215826Z 2025-12-04T09:37:33.9215936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9216135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9216236Z res = mod(**inputs) 2025-12-04T09:37:33.9216482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1295, in forward 2025-12-04T09:37:33.9216575Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:37:33.9216821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 752, in forward 2025-12-04T09:37:33.9216929Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:37:33.9217171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 741, in forward 2025-12-04T09:37:33.9217269Z hidden_states = self.transform(hidden_states) 2025-12-04T09:37:33.9217516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 717, in forward 2025-12-04T09:37:33.9217601Z hidden_states = self.dense(hidden_states) 2025-12-04T09:37:33.9217607Z 2025-12-04T09:37:33.9217707Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9217903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9217972Z res = mod(**inputs) 2025-12-04T09:37:33.9218221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1295, in forward 2025-12-04T09:37:33.9218315Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:37:33.9218562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 752, in forward 2025-12-04T09:37:33.9218669Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:37:33.9218922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 742, in forward 2025-12-04T09:37:33.9219011Z hidden_states = self.decoder(hidden_states) 2025-12-04T09:37:33.9219018Z 2025-12-04T09:37:33.9219119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:37:33.9219324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:37:33.9219386Z res = mod(**inputs) 2025-12-04T09:37:33.9219639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1300, in forward 2025-12-04T09:37:33.9219829Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:37:33.9219833Z 2025-12-04T09:37:43.8164548Z Compilation time (from dynamo_timed): 16.407892412 2025-12-04T09:37:43.8231820Z pass 2025-12-04T09:37:43.8232252Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:37:43.8233127Z TIMING: _recursive_pre_grad_passes:0.00645 _recursive_joint_graph_passes:0.64171 _recursive_post_grad_passes:0.0635 async_compile.wait:0.75901 code_gen:9.42855 inductor_compile:10.67694 backend_compile:13.65691 gc:0.00091 entire_frame_compile:16.40789 total_wall_time:16.40789 2025-12-04T09:37:43.8234351Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:7149 | FakeTensor.__torch_dispatch__:4342 | ProxyTorchDispatchMode.__torch_dispatch__:1972 2025-12-04T09:37:43.8234883Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-12-04T09:37:46.1563171Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:37:46.1568280Z import pynvml # type: ignore[import] 2025-12-04T09:37:49.5107884Z 2025-12-04T09:38:05.6391092Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:38:05.6394053Z loading model: 0it [00:16, ?it/s] 2025-12-04T09:38:05.6425420Z cpu eval BlenderbotForCausalLM 2025-12-04T09:38:05.6606750Z Compilation time (from dynamo_timed): 0 2025-12-04T09:38:05.6607042Z pass_due_to_skip 2025-12-04T09:38:05.6607391Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:05.6607736Z TIMING: total_wall_time:0 2025-12-04T09:38:05.6607933Z STATS: call_* op count: 0 2025-12-04T09:38:05.6608198Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-12-04T09:38:07.6372605Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:38:07.6373523Z import pynvml # type: ignore[import] 2025-12-04T09:38:10.9979794Z 2025-12-04T09:38:18.5895427Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:38:18.5897120Z loading model: 0it [00:07, ?it/s] 2025-12-04T09:38:18.5924510Z cpu eval DebertaV2ForMaskedLM 2025-12-04T09:38:18.6116428Z Compilation time (from dynamo_timed): 0 2025-12-04T09:38:18.6116740Z pass_due_to_skip 2025-12-04T09:38:18.6117242Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:18.6117741Z TIMING: total_wall_time:0 2025-12-04T09:38:18.6118064Z STATS: call_* op count: 0 2025-12-04T09:38:18.6118452Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-12-04T09:38:20.6394104Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:38:20.6394994Z import pynvml # type: ignore[import] 2025-12-04T09:38:24.2345402Z 2025-12-04T09:38:24.8754947Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:38:24.8755561Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:38:24.8765394Z cpu eval DistilBertForMaskedLM 2025-12-04T09:38:25.0467770Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:25.1035712Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:25.1599397Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:29.8803411Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8803720Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8807656Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8807945Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8808170Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8808381Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8808672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8809123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8809476Z res = mod(**inputs) 2025-12-04T09:38:29.8810237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8810710Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8811163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8811742Z return self.transformer( 2025-12-04T09:38:29.8812202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8812749Z layer_outputs = layer_module( 2025-12-04T09:38:29.8813152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8813588Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8814107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8814528Z sa_output = self.attention( 2025-12-04T09:38:29.8814951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:38:29.8815458Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:38:29.8815660Z 2025-12-04T09:38:29.8815775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8816164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8816511Z res = mod(**inputs) 2025-12-04T09:38:29.8816923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8817368Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8817813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8818258Z return self.transformer( 2025-12-04T09:38:29.8818685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8819088Z layer_outputs = layer_module( 2025-12-04T09:38:29.8819436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8819819Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8820258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8820672Z sa_output = self.attention( 2025-12-04T09:38:29.8821069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:38:29.8821613Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.8821793Z 2025-12-04T09:38:29.8821899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8822259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8822583Z res = mod(**inputs) 2025-12-04T09:38:29.8822966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8823379Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8823787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8824202Z return self.transformer( 2025-12-04T09:38:29.8824592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8825010Z layer_outputs = layer_module( 2025-12-04T09:38:29.8825448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8825818Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8826236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8826650Z sa_output = self.attention( 2025-12-04T09:38:29.8827055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:38:29.8827511Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.8827749Z 2025-12-04T09:38:29.8827841Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8828081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8828479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8828797Z res = mod(**inputs) 2025-12-04T09:38:29.8829190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8829607Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8830016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8830415Z return self.transformer( 2025-12-04T09:38:29.8830789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8831178Z layer_outputs = layer_module( 2025-12-04T09:38:29.8831498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8831859Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8832257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8832643Z sa_output = self.attention( 2025-12-04T09:38:29.8833014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:38:29.8833472Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:29.8833650Z 2025-12-04T09:38:29.8833760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8834106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8834412Z res = mod(**inputs) 2025-12-04T09:38:29.8834784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8835188Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8835574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8835971Z return self.transformer( 2025-12-04T09:38:29.8836357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8836747Z layer_outputs = layer_module( 2025-12-04T09:38:29.8837079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8837434Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8837832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8838226Z sa_output = self.attention( 2025-12-04T09:38:29.8838610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:38:29.8839036Z attn_output = self.out_lin(attn_output) 2025-12-04T09:38:29.8839830Z 2025-12-04T09:38:29.8839952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8840285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8840590Z res = mod(**inputs) 2025-12-04T09:38:29.8840957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8841345Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8841728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8842137Z return self.transformer( 2025-12-04T09:38:29.8842518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8842951Z layer_outputs = layer_module( 2025-12-04T09:38:29.8843292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8843647Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8844052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8844484Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8844925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8845457Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8845967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8846352Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8846763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:38:29.8847215Z x = self.lin1(input) 2025-12-04T09:38:29.8847320Z 2025-12-04T09:38:29.8847420Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8847769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8848083Z res = mod(**inputs) 2025-12-04T09:38:29.8848455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8848852Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8849256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8849671Z return self.transformer( 2025-12-04T09:38:29.8850069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8850473Z layer_outputs = layer_module( 2025-12-04T09:38:29.8850818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8851184Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8851760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8852270Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8852767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8853360Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8853963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8854376Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8854793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:38:29.8855210Z x = self.activation(x) 2025-12-04T09:38:29.8855536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:29.8855884Z return self.act(input) 2025-12-04T09:38:29.8855995Z 2025-12-04T09:38:29.8856111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8856465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8856806Z res = mod(**inputs) 2025-12-04T09:38:29.8857215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8857626Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8858020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8858423Z return self.transformer( 2025-12-04T09:38:29.8858811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8859214Z layer_outputs = layer_module( 2025-12-04T09:38:29.8859550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8859908Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8860314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8860748Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8861190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8861713Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8862215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8862593Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8862996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:38:29.8863395Z x = self.lin2(x) 2025-12-04T09:38:29.8863491Z 2025-12-04T09:38:29.8863600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8863940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8864256Z res = mod(**inputs) 2025-12-04T09:38:29.8864634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8865027Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8865425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8865821Z return self.transformer( 2025-12-04T09:38:29.8866204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8866596Z layer_outputs = layer_module( 2025-12-04T09:38:29.8866935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8867290Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8867688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8868117Z sa_output = self.attention( 2025-12-04T09:38:29.8868513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:38:29.8868979Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:38:29.8869161Z 2025-12-04T09:38:29.8869265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8869622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8869939Z res = mod(**inputs) 2025-12-04T09:38:29.8870317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8870715Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8871146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8871552Z return self.transformer( 2025-12-04T09:38:29.8871929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8872648Z layer_outputs = layer_module( 2025-12-04T09:38:29.8872994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8873357Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8873759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8874160Z sa_output = self.attention( 2025-12-04T09:38:29.8874548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:38:29.8874999Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.8875168Z 2025-12-04T09:38:29.8875271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8875620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8875927Z res = mod(**inputs) 2025-12-04T09:38:29.8876290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8876690Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8877091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8877490Z return self.transformer( 2025-12-04T09:38:29.8877867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8878271Z layer_outputs = layer_module( 2025-12-04T09:38:29.8878613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8878967Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8879366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8879759Z sa_output = self.attention( 2025-12-04T09:38:29.8880146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:38:29.8880589Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.8880765Z 2025-12-04T09:38:29.8880842Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8881068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8881405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8881705Z res = mod(**inputs) 2025-12-04T09:38:29.8882227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8882639Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8883171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8883550Z return self.transformer( 2025-12-04T09:38:29.8883926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8884319Z layer_outputs = layer_module( 2025-12-04T09:38:29.8884651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8885093Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8885493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8885890Z sa_output = self.attention( 2025-12-04T09:38:29.8886266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:38:29.8886718Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:29.8886902Z 2025-12-04T09:38:29.8886998Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8887333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8887636Z res = mod(**inputs) 2025-12-04T09:38:29.8888006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8888406Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8888787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8889183Z return self.transformer( 2025-12-04T09:38:29.8889555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8889940Z layer_outputs = layer_module( 2025-12-04T09:38:29.8890262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8890616Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8891018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8891466Z sa_output = self.attention( 2025-12-04T09:38:29.8891856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:38:29.8892284Z attn_output = self.out_lin(attn_output) 2025-12-04T09:38:29.8892430Z 2025-12-04T09:38:29.8892547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8892914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8893251Z res = mod(**inputs) 2025-12-04T09:38:29.8893657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8894091Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8894504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8894905Z return self.transformer( 2025-12-04T09:38:29.8895355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8895796Z layer_outputs = layer_module( 2025-12-04T09:38:29.8896200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8896593Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8897042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8897520Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8897998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8898577Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8899133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8899596Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8900037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:38:29.8900468Z x = self.lin1(input) 2025-12-04T09:38:29.8900581Z 2025-12-04T09:38:29.8900695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8901062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8901401Z res = mod(**inputs) 2025-12-04T09:38:29.8901801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8902231Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8902619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8903005Z return self.transformer( 2025-12-04T09:38:29.8903390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8903768Z layer_outputs = layer_module( 2025-12-04T09:38:29.8904096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8904438Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8904829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8905241Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8905658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8906165Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8906654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8907029Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8907417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:38:29.8907807Z x = self.activation(x) 2025-12-04T09:38:29.8908109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:29.8908433Z return self.act(input) 2025-12-04T09:38:29.8908535Z 2025-12-04T09:38:29.8908640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8908978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8909271Z res = mod(**inputs) 2025-12-04T09:38:29.8909629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8910016Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8910418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8910808Z return self.transformer( 2025-12-04T09:38:29.8911180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8911574Z layer_outputs = layer_module( 2025-12-04T09:38:29.8911903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8912255Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8912658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8913136Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8913651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8914194Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8914700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8915082Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8915489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:38:29.8915905Z x = self.lin2(x) 2025-12-04T09:38:29.8916002Z 2025-12-04T09:38:29.8916108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8916459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8916783Z res = mod(**inputs) 2025-12-04T09:38:29.8917177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8917587Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8917978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8918400Z return self.transformer( 2025-12-04T09:38:29.8918790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8919174Z layer_outputs = layer_module( 2025-12-04T09:38:29.8919519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8919878Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8920291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8920701Z sa_output = self.attention( 2025-12-04T09:38:29.8921100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:38:29.8921562Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:38:29.8921736Z 2025-12-04T09:38:29.8921844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8922184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8922507Z res = mod(**inputs) 2025-12-04T09:38:29.8922870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8923263Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8923653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8924081Z return self.transformer( 2025-12-04T09:38:29.8924458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8924842Z layer_outputs = layer_module( 2025-12-04T09:38:29.8925171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8925518Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8925906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8926293Z sa_output = self.attention( 2025-12-04T09:38:29.8926672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:38:29.8927195Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.8927362Z 2025-12-04T09:38:29.8927461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8927812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8928114Z res = mod(**inputs) 2025-12-04T09:38:29.8928478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8928864Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8929253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8929644Z return self.transformer( 2025-12-04T09:38:29.8930018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8930426Z layer_outputs = layer_module( 2025-12-04T09:38:29.8930763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8931119Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8931602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8932042Z sa_output = self.attention( 2025-12-04T09:38:29.8932468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:38:29.8932962Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.8933151Z 2025-12-04T09:38:29.8933246Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.8933483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8933847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8934150Z res = mod(**inputs) 2025-12-04T09:38:29.8934544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8934985Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8935414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8935846Z return self.transformer( 2025-12-04T09:38:29.8936265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8936708Z layer_outputs = layer_module( 2025-12-04T09:38:29.8937067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8937518Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8937967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8938470Z sa_output = self.attention( 2025-12-04T09:38:29.8938894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:38:29.8939407Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:29.8939611Z 2025-12-04T09:38:29.8939720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8940099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8940419Z res = mod(**inputs) 2025-12-04T09:38:29.8940789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8941192Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8941615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8942091Z return self.transformer( 2025-12-04T09:38:29.8942479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8942879Z layer_outputs = layer_module( 2025-12-04T09:38:29.8943216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8943575Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8943987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8944392Z sa_output = self.attention( 2025-12-04T09:38:29.8944770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:38:29.8945183Z attn_output = self.out_lin(attn_output) 2025-12-04T09:38:29.8945317Z 2025-12-04T09:38:29.8945427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8945767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8946080Z res = mod(**inputs) 2025-12-04T09:38:29.8946458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8946860Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8947248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8947649Z return self.transformer( 2025-12-04T09:38:29.8948038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8948442Z layer_outputs = layer_module( 2025-12-04T09:38:29.8948782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8949135Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8949550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8949984Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8950418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8950941Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8951447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8951835Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8952274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:38:29.8952673Z x = self.lin1(input) 2025-12-04T09:38:29.8952774Z 2025-12-04T09:38:29.8952879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8953210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8953517Z res = mod(**inputs) 2025-12-04T09:38:29.8953880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8954264Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8954653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8955044Z return self.transformer( 2025-12-04T09:38:29.8955480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8955875Z layer_outputs = layer_module( 2025-12-04T09:38:29.8956211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8956558Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8956953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8957383Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8957809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8958319Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8958808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8959189Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8959585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:38:29.8959977Z x = self.activation(x) 2025-12-04T09:38:29.8960285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:29.8960611Z return self.act(input) 2025-12-04T09:38:29.8960714Z 2025-12-04T09:38:29.8960821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8961157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8961453Z res = mod(**inputs) 2025-12-04T09:38:29.8961815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8962209Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8962595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8962985Z return self.transformer( 2025-12-04T09:38:29.8963365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8963759Z layer_outputs = layer_module( 2025-12-04T09:38:29.8964095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8964442Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8964826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.8965234Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.8965681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.8966179Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.8966652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.8967014Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.8967402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:38:29.8967786Z x = self.lin2(x) 2025-12-04T09:38:29.8967881Z 2025-12-04T09:38:29.8967996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8968317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8968657Z res = mod(**inputs) 2025-12-04T09:38:29.8969029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8969423Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8969809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8970199Z return self.transformer( 2025-12-04T09:38:29.8970587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8970972Z layer_outputs = layer_module( 2025-12-04T09:38:29.8971414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8971832Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8972473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8972921Z sa_output = self.attention( 2025-12-04T09:38:29.8973359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:38:29.8973813Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:38:29.8973981Z 2025-12-04T09:38:29.8974084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8974409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8974708Z res = mod(**inputs) 2025-12-04T09:38:29.8975064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8975445Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8975824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8976202Z return self.transformer( 2025-12-04T09:38:29.8976568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.8976952Z layer_outputs = layer_module( 2025-12-04T09:38:29.8977283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.8991067Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.8991575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.8991992Z sa_output = self.attention( 2025-12-04T09:38:29.8992437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:38:29.8992913Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.8993099Z 2025-12-04T09:38:29.8993381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.8997342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.8997665Z res = mod(**inputs) 2025-12-04T09:38:29.8998043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.8998451Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.8998855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.8999259Z return self.transformer( 2025-12-04T09:38:29.8999633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9000092Z layer_outputs = layer_module( 2025-12-04T09:38:29.9000428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9000777Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9001214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9001602Z sa_output = self.attention( 2025-12-04T09:38:29.9001989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:38:29.9002432Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.9002604Z 2025-12-04T09:38:29.9002684Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.9002915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9003261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9003575Z res = mod(**inputs) 2025-12-04T09:38:29.9003973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9004383Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9004783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9005199Z return self.transformer( 2025-12-04T09:38:29.9005600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9006016Z layer_outputs = layer_module( 2025-12-04T09:38:29.9006358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9006731Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9007140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9007539Z sa_output = self.attention( 2025-12-04T09:38:29.9007929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:38:29.9008380Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:29.9008569Z 2025-12-04T09:38:29.9008672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9009025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9009337Z res = mod(**inputs) 2025-12-04T09:38:29.9009702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9010110Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9010494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9010921Z return self.transformer( 2025-12-04T09:38:29.9011447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9011956Z layer_outputs = layer_module( 2025-12-04T09:38:29.9012332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9012706Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9013101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9013488Z sa_output = self.attention( 2025-12-04T09:38:29.9013860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:38:29.9014281Z attn_output = self.out_lin(attn_output) 2025-12-04T09:38:29.9014419Z 2025-12-04T09:38:29.9014522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9014862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9015164Z res = mod(**inputs) 2025-12-04T09:38:29.9015532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9015928Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9016304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9016695Z return self.transformer( 2025-12-04T09:38:29.9017077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9017467Z layer_outputs = layer_module( 2025-12-04T09:38:29.9017795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9018146Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9018544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9018971Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9019385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9019897Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9020397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9020772Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9021162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:38:29.9021554Z x = self.lin1(input) 2025-12-04T09:38:29.9021658Z 2025-12-04T09:38:29.9021769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9022107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9022406Z res = mod(**inputs) 2025-12-04T09:38:29.9022762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9023141Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9023512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9023891Z return self.transformer( 2025-12-04T09:38:29.9024257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9024666Z layer_outputs = layer_module( 2025-12-04T09:38:29.9025033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9025370Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9025764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9026192Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9026606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9027115Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9027622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9028005Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9028395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:38:29.9028790Z x = self.activation(x) 2025-12-04T09:38:29.9029103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:29.9029428Z return self.act(input) 2025-12-04T09:38:29.9029533Z 2025-12-04T09:38:29.9029631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9029971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9030277Z res = mod(**inputs) 2025-12-04T09:38:29.9030633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9031030Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9031423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9031813Z return self.transformer( 2025-12-04T09:38:29.9032180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9032569Z layer_outputs = layer_module( 2025-12-04T09:38:29.9032898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9033243Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9033626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9034056Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9034490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9035007Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9035509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9035895Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9036301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:38:29.9036684Z x = self.lin2(x) 2025-12-04T09:38:29.9036787Z 2025-12-04T09:38:29.9036886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9037231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9037537Z res = mod(**inputs) 2025-12-04T09:38:29.9038008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9038402Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9038814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9039200Z return self.transformer( 2025-12-04T09:38:29.9039581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9039979Z layer_outputs = layer_module( 2025-12-04T09:38:29.9040316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9040676Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9041079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9041484Z sa_output = self.attention( 2025-12-04T09:38:29.9041865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:38:29.9042303Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:38:29.9042483Z 2025-12-04T09:38:29.9042583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9042930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9043232Z res = mod(**inputs) 2025-12-04T09:38:29.9043604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9043986Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9044365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9044740Z return self.transformer( 2025-12-04T09:38:29.9045112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9045496Z layer_outputs = layer_module( 2025-12-04T09:38:29.9045821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9046169Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9046568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9046961Z sa_output = self.attention( 2025-12-04T09:38:29.9047335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:38:29.9047777Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.9047954Z 2025-12-04T09:38:29.9048055Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9048403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9048704Z res = mod(**inputs) 2025-12-04T09:38:29.9049070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9049466Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9049845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9050237Z return self.transformer( 2025-12-04T09:38:29.9050612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9051002Z layer_outputs = layer_module( 2025-12-04T09:38:29.9051427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9051868Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9052340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9052785Z sa_output = self.attention( 2025-12-04T09:38:29.9053164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:38:29.9053612Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.9053784Z 2025-12-04T09:38:29.9053870Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.9054093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9054438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9054773Z res = mod(**inputs) 2025-12-04T09:38:29.9055153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9055553Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9055953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9056361Z return self.transformer( 2025-12-04T09:38:29.9056732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9057121Z layer_outputs = layer_module( 2025-12-04T09:38:29.9057450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9057801Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9058195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9058591Z sa_output = self.attention( 2025-12-04T09:38:29.9058972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:38:29.9059424Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:29.9059599Z 2025-12-04T09:38:29.9059697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9060039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9060346Z res = mod(**inputs) 2025-12-04T09:38:29.9060703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9061090Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9061474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9061867Z return self.transformer( 2025-12-04T09:38:29.9062239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9062630Z layer_outputs = layer_module( 2025-12-04T09:38:29.9062965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9063318Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9063715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9064104Z sa_output = self.attention( 2025-12-04T09:38:29.9064483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:38:29.9064878Z attn_output = self.out_lin(attn_output) 2025-12-04T09:38:29.9065019Z 2025-12-04T09:38:29.9065117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9065487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9065817Z res = mod(**inputs) 2025-12-04T09:38:29.9066176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9066576Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9066966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9067355Z return self.transformer( 2025-12-04T09:38:29.9067725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9068116Z layer_outputs = layer_module( 2025-12-04T09:38:29.9068468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9068811Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9069219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9069655Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9070088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9070606Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9071112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9071515Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9071920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:38:29.9072485Z x = self.lin1(input) 2025-12-04T09:38:29.9072606Z 2025-12-04T09:38:29.9072713Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9073078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9073394Z res = mod(**inputs) 2025-12-04T09:38:29.9073782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9074194Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9074613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9075023Z return self.transformer( 2025-12-04T09:38:29.9075428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9075850Z layer_outputs = layer_module( 2025-12-04T09:38:29.9076211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9076577Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9076992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9077441Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9077874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9078411Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9078921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9079317Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9079783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:38:29.9080223Z x = self.activation(x) 2025-12-04T09:38:29.9080548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:29.9080882Z return self.act(input) 2025-12-04T09:38:29.9080992Z 2025-12-04T09:38:29.9081093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9081445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9081762Z res = mod(**inputs) 2025-12-04T09:38:29.9082134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9082567Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9082965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9083364Z return self.transformer( 2025-12-04T09:38:29.9083747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9084147Z layer_outputs = layer_module( 2025-12-04T09:38:29.9084488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9084834Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9085236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9085668Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9086099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9086625Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9087125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9087512Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9087913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:38:29.9088305Z x = self.lin2(x) 2025-12-04T09:38:29.9088401Z 2025-12-04T09:38:29.9088501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9088848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9089160Z res = mod(**inputs) 2025-12-04T09:38:29.9089528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9089933Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9090333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9090728Z return self.transformer( 2025-12-04T09:38:29.9091107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9091567Z layer_outputs = layer_module( 2025-12-04T09:38:29.9091913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9092278Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9092690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9093094Z sa_output = self.attention( 2025-12-04T09:38:29.9093518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T09:38:29.9093983Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T09:38:29.9094168Z 2025-12-04T09:38:29.9094270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9094627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9094946Z res = mod(**inputs) 2025-12-04T09:38:29.9095325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9095775Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9096211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9096617Z return self.transformer( 2025-12-04T09:38:29.9096992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9097384Z layer_outputs = layer_module( 2025-12-04T09:38:29.9097717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9098060Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9098459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9098850Z sa_output = self.attention( 2025-12-04T09:38:29.9099232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T09:38:29.9099664Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.9099836Z 2025-12-04T09:38:29.9099933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9100274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9100577Z res = mod(**inputs) 2025-12-04T09:38:29.9100934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9101324Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9101710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9102091Z return self.transformer( 2025-12-04T09:38:29.9102467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9102856Z layer_outputs = layer_module( 2025-12-04T09:38:29.9103187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9103527Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9103924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9104313Z sa_output = self.attention( 2025-12-04T09:38:29.9104691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T09:38:29.9105129Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T09:38:29.9105304Z 2025-12-04T09:38:29.9105381Z cudagraph partition due to non gpu ops 2025-12-04T09:38:29.9105609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9105939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9106244Z res = mod(**inputs) 2025-12-04T09:38:29.9106652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9107066Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9107445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9107830Z return self.transformer( 2025-12-04T09:38:29.9108204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9108586Z layer_outputs = layer_module( 2025-12-04T09:38:29.9108923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9109289Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9109683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9110079Z sa_output = self.attention( 2025-12-04T09:38:29.9110459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T09:38:29.9110907Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:29.9111079Z 2025-12-04T09:38:29.9111181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9111508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9111808Z res = mod(**inputs) 2025-12-04T09:38:29.9112168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9112547Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9112930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9113317Z return self.transformer( 2025-12-04T09:38:29.9113690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9114072Z layer_outputs = layer_module( 2025-12-04T09:38:29.9114399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9114742Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9115125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T09:38:29.9115511Z sa_output = self.attention( 2025-12-04T09:38:29.9115887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T09:38:29.9116280Z attn_output = self.out_lin(attn_output) 2025-12-04T09:38:29.9116411Z 2025-12-04T09:38:29.9116508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9116848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9117156Z res = mod(**inputs) 2025-12-04T09:38:29.9117518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9117896Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9118277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9118661Z return self.transformer( 2025-12-04T09:38:29.9119024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9119411Z layer_outputs = layer_module( 2025-12-04T09:38:29.9119740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9120114Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9120517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9120939Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9121358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9121866Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9122333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9122701Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9123100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T09:38:29.9123478Z x = self.lin1(input) 2025-12-04T09:38:29.9123588Z 2025-12-04T09:38:29.9123689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9124026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9124332Z res = mod(**inputs) 2025-12-04T09:38:29.9124687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9125074Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9125457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9125849Z return self.transformer( 2025-12-04T09:38:29.9126219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9126617Z layer_outputs = layer_module( 2025-12-04T09:38:29.9126954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9127308Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9127704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9128139Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9128567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9129081Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9129570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9129959Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9130362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T09:38:29.9130762Z x = self.activation(x) 2025-12-04T09:38:29.9131078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:38:29.9131532Z return self.act(input) 2025-12-04T09:38:29.9131651Z 2025-12-04T09:38:29.9131772Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9132150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9132496Z res = mod(**inputs) 2025-12-04T09:38:29.9132887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T09:38:29.9133314Z dlbrt_output = self.distilbert( 2025-12-04T09:38:29.9133795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T09:38:29.9134200Z return self.transformer( 2025-12-04T09:38:29.9134583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T09:38:29.9134962Z layer_outputs = layer_module( 2025-12-04T09:38:29.9135289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:29.9135632Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:29.9136025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T09:38:29.9136437Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T09:38:29.9136852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T09:38:29.9137385Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T09:38:29.9137873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:38:29.9138241Z return forward_fn(*input_tensors) 2025-12-04T09:38:29.9138626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T09:38:29.9139009Z x = self.lin2(x) 2025-12-04T09:38:29.9139102Z 2025-12-04T09:38:29.9139200Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9139539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9139894Z res = mod(**inputs) 2025-12-04T09:38:29.9140258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 824, in forward 2025-12-04T09:38:29.9140720Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-12-04T09:38:29.9140932Z 2025-12-04T09:38:29.9141027Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9141361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9141657Z res = mod(**inputs) 2025-12-04T09:38:29.9142010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 827, in forward 2025-12-04T09:38:29.9142498Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-12-04T09:38:29.9142722Z 2025-12-04T09:38:29.9142825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:29.9143155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:38:29.9143447Z res = mod(**inputs) 2025-12-04T09:38:29.9143807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 831, in forward 2025-12-04T09:38:29.9144308Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-12-04T09:38:29.9144537Z 2025-12-04T09:38:38.8929405Z Compilation time (from dynamo_timed): 13.236925706 2025-12-04T09:38:38.8953792Z pass 2025-12-04T09:38:38.8954332Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:38.8955349Z TIMING: _recursive_pre_grad_passes:0.00527 _recursive_joint_graph_passes:0.25206 _recursive_post_grad_passes:0.04436 async_compile.wait:0.78638 code_gen:9.07447 inductor_compile:9.77713 backend_compile:11.53614 gc:0.00074 entire_frame_compile:13.23693 total_wall_time:13.23693 2025-12-04T09:38:38.8956388Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:3980 | FakeTensor.__torch_dispatch__:2344 | ProxyTorchDispatchMode.__torch_dispatch__:1053 2025-12-04T09:38:38.8957197Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-12-04T09:38:41.1914139Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:38:41.1915258Z import pynvml # type: ignore[import] 2025-12-04T09:38:44.5517464Z 2025-12-04T09:38:45.4760638Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-12-04T09:38:45.4761377Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-12-04T09:38:45.4980865Z 2025-12-04T09:38:45.4987273Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:38:45.4989269Z cpu eval DistillGPT2 2025-12-04T09:38:45.9153593Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:46.1084137Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:46.3002243Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:38:52.7179722Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7183327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7183786Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7184272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:38:52.7184701Z causal_mask = create_causal_mask( 2025-12-04T09:38:52.7185106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:38:52.7185633Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:38:52.7186164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:38:52.7186647Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:38:52.7187175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-12-04T09:38:52.7187706Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-12-04T09:38:52.7187935Z 2025-12-04T09:38:52.7188031Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7188326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7188777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7189184Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7189577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7189958Z outputs = block( 2025-12-04T09:38:52.7190293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7190669Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7191050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7191427Z return func(*args, **kwargs) 2025-12-04T09:38:52.7191801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7192197Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7192941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7193394Z return func(*args, **kwargs) 2025-12-04T09:38:52.7193766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:38:52.7194258Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:38:52.7194720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7195123Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7195298Z 2025-12-04T09:38:52.7195412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7195825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7196280Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7196672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:38:52.7197151Z causal_mask = create_causal_mask( 2025-12-04T09:38:52.7197533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:38:52.7198085Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:38:52.7198609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:38:52.7199059Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:38:52.7199514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-12-04T09:38:52.7200010Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-12-04T09:38:52.7200222Z 2025-12-04T09:38:52.7200312Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7200557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7200970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7201359Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7201743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:38:52.7202113Z causal_mask = create_causal_mask( 2025-12-04T09:38:52.7202589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:38:52.7203076Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:38:52.7203591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:38:52.7204037Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:38:52.7204492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T09:38:52.7204931Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T09:38:52.7205096Z 2025-12-04T09:38:52.7205201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7205616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7206015Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7206406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:38:52.7206810Z causal_mask = create_causal_mask( 2025-12-04T09:38:52.7207245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:38:52.7207789Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:38:52.7208329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:38:52.7208790Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:38:52.7209262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T09:38:52.7209718Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T09:38:52.7209887Z 2025-12-04T09:38:52.7210000Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7210250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7210688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7211109Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7211738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7212163Z outputs = block( 2025-12-04T09:38:52.7212522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7212918Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7213301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7213664Z return func(*args, **kwargs) 2025-12-04T09:38:52.7214027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7214412Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7214790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7215153Z return func(*args, **kwargs) 2025-12-04T09:38:52.7215520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:38:52.7215919Z attn_output, attn_weights = attention_interface( 2025-12-04T09:38:52.7216367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:38:52.7216858Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:52.7217036Z 2025-12-04T09:38:52.7217143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7217538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7217929Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7218317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7218680Z outputs = block( 2025-12-04T09:38:52.7219006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7219370Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7219747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7220111Z return func(*args, **kwargs) 2025-12-04T09:38:52.7220478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7220870Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7221341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7221730Z return func(*args, **kwargs) 2025-12-04T09:38:52.7222096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:38:52.7222485Z attn_output = self.c_proj(attn_output) 2025-12-04T09:38:52.7222834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7223233Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7223411Z 2025-12-04T09:38:52.7223514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7223926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7224332Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7224725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7225103Z outputs = block( 2025-12-04T09:38:52.7225431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7225790Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7226177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7226554Z return func(*args, **kwargs) 2025-12-04T09:38:52.7226920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7227335Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7227748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:38:52.7228141Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:38:52.7228492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7228892Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7229067Z 2025-12-04T09:38:52.7229177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7229645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7230041Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7230430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7230806Z outputs = block( 2025-12-04T09:38:52.7231113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7231467Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7231836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7232198Z return func(*args, **kwargs) 2025-12-04T09:38:52.7232551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7232960Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7233368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:38:52.7233744Z hidden_states = self.act(hidden_states) 2025-12-04T09:38:52.7234098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:38:52.7234543Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:38:52.7234769Z 2025-12-04T09:38:52.7234916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7235315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7235721Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7236095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7236461Z outputs = block( 2025-12-04T09:38:52.7236784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7237148Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7237536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7237910Z return func(*args, **kwargs) 2025-12-04T09:38:52.7238272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7238669Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7239064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:38:52.7239438Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:38:52.7239787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7240171Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7240334Z 2025-12-04T09:38:52.7240439Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7240832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7241214Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7241590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7241942Z outputs = block( 2025-12-04T09:38:52.7242255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7242610Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7242978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7243333Z return func(*args, **kwargs) 2025-12-04T09:38:52.7243690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7244130Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7244507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7244860Z return func(*args, **kwargs) 2025-12-04T09:38:52.7245218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:38:52.7245702Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:38:52.7246129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7246499Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7246663Z 2025-12-04T09:38:52.7246740Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7246965Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7247355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7247741Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7248147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7248519Z outputs = block( 2025-12-04T09:38:52.7248854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7249216Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7249592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7249959Z return func(*args, **kwargs) 2025-12-04T09:38:52.7250326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7250725Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7251112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7251641Z return func(*args, **kwargs) 2025-12-04T09:38:52.7252021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:38:52.7252431Z attn_output, attn_weights = attention_interface( 2025-12-04T09:38:52.7252877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:38:52.7253366Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:52.7253558Z 2025-12-04T09:38:52.7253660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7254077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7254467Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7254846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7255207Z outputs = block( 2025-12-04T09:38:52.7255523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7255869Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7256245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7256632Z return func(*args, **kwargs) 2025-12-04T09:38:52.7256991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7257387Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7257783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7258144Z return func(*args, **kwargs) 2025-12-04T09:38:52.7258495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:38:52.7258874Z attn_output = self.c_proj(attn_output) 2025-12-04T09:38:52.7259225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7259608Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7259774Z 2025-12-04T09:38:52.7259874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7260276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7260658Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7261029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7261392Z outputs = block( 2025-12-04T09:38:52.7261712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7262141Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7262508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7262917Z return func(*args, **kwargs) 2025-12-04T09:38:52.7263273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7263666Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7264065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:38:52.7264446Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:38:52.7264793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7265200Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7265374Z 2025-12-04T09:38:52.7265475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7265875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7266256Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7266625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7266984Z outputs = block( 2025-12-04T09:38:52.7267297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7267641Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7268008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7268375Z return func(*args, **kwargs) 2025-12-04T09:38:52.7268731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7269124Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7269517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:38:52.7269936Z hidden_states = self.act(hidden_states) 2025-12-04T09:38:52.7270288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:38:52.7270720Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:38:52.7270952Z 2025-12-04T09:38:52.7271054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7271459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7271836Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7272215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7272837Z outputs = block( 2025-12-04T09:38:52.7273155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7273503Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7273871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7274237Z return func(*args, **kwargs) 2025-12-04T09:38:52.7274599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7274993Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7275395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:38:52.7275870Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:38:52.7276249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7276638Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7276812Z 2025-12-04T09:38:52.7276912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7277311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7277690Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7278064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7278429Z outputs = block( 2025-12-04T09:38:52.7278761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7279101Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7279464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7279822Z return func(*args, **kwargs) 2025-12-04T09:38:52.7280163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7280540Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7280907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7281257Z return func(*args, **kwargs) 2025-12-04T09:38:52.7281596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:38:52.7282071Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:38:52.7282523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7282915Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7283081Z 2025-12-04T09:38:52.7283162Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7283397Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7283800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7284179Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7284559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7284924Z outputs = block( 2025-12-04T09:38:52.7285239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7285587Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7285960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7286329Z return func(*args, **kwargs) 2025-12-04T09:38:52.7286682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7287069Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7287450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7287826Z return func(*args, **kwargs) 2025-12-04T09:38:52.7288195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:38:52.7288601Z attn_output, attn_weights = attention_interface( 2025-12-04T09:38:52.7289077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:38:52.7289579Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:52.7289761Z 2025-12-04T09:38:52.7289864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7290280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7290678Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7291062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7291499Z outputs = block( 2025-12-04T09:38:52.7291847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7292254Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7292671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7293083Z return func(*args, **kwargs) 2025-12-04T09:38:52.7293497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7293919Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7294315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7294682Z return func(*args, **kwargs) 2025-12-04T09:38:52.7295051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:38:52.7295430Z attn_output = self.c_proj(attn_output) 2025-12-04T09:38:52.7295807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7296225Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7296405Z 2025-12-04T09:38:52.7296522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7296950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7297431Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7297817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7298184Z outputs = block( 2025-12-04T09:38:52.7298497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7298857Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7299232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7299597Z return func(*args, **kwargs) 2025-12-04T09:38:52.7299967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7300376Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7300779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:38:52.7301185Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:38:52.7301538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7301930Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7302099Z 2025-12-04T09:38:52.7302199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7302606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7303002Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7303422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7303803Z outputs = block( 2025-12-04T09:38:52.7304127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7304490Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7304871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7305239Z return func(*args, **kwargs) 2025-12-04T09:38:52.7305613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7306024Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7306443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:38:52.7306837Z hidden_states = self.act(hidden_states) 2025-12-04T09:38:52.7307191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:38:52.7307649Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:38:52.7307878Z 2025-12-04T09:38:52.7307981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7308397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7308788Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7309172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7309540Z outputs = block( 2025-12-04T09:38:52.7309850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7310213Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7310592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7310952Z return func(*args, **kwargs) 2025-12-04T09:38:52.7311312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7311708Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7312092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:38:52.7312476Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:38:52.7312827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7313217Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7313380Z 2025-12-04T09:38:52.7313483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7313889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7314272Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7314638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7314994Z outputs = block( 2025-12-04T09:38:52.7315305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7315651Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7316008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7316372Z return func(*args, **kwargs) 2025-12-04T09:38:52.7316758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:38:52.7317186Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:38:52.7317348Z 2025-12-04T09:38:52.7317448Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7317993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7318395Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7318775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7319150Z outputs = block( 2025-12-04T09:38:52.7319470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7319850Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7320213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7320577Z return func(*args, **kwargs) 2025-12-04T09:38:52.7320940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7321325Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7321712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7322079Z return func(*args, **kwargs) 2025-12-04T09:38:52.7322447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:38:52.7322928Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:38:52.7323390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7323776Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7323945Z 2025-12-04T09:38:52.7324034Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7324263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7324675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7325072Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7325457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7325842Z outputs = block( 2025-12-04T09:38:52.7326180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7326562Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7326942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7327322Z return func(*args, **kwargs) 2025-12-04T09:38:52.7327684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7328087Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7328493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7328878Z return func(*args, **kwargs) 2025-12-04T09:38:52.7329265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:38:52.7329708Z attn_output, attn_weights = attention_interface( 2025-12-04T09:38:52.7330175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:38:52.7330717Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:52.7330932Z 2025-12-04T09:38:52.7331049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7331563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7332004Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7332436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7332830Z outputs = block( 2025-12-04T09:38:52.7333166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7333528Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7333902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7334289Z return func(*args, **kwargs) 2025-12-04T09:38:52.7334661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7335054Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7335437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7335798Z return func(*args, **kwargs) 2025-12-04T09:38:52.7336161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:38:52.7336547Z attn_output = self.c_proj(attn_output) 2025-12-04T09:38:52.7336891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7337287Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7337461Z 2025-12-04T09:38:52.7337566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7337976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7338359Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7338744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7339114Z outputs = block( 2025-12-04T09:38:52.7339434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7339793Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7340165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7340534Z return func(*args, **kwargs) 2025-12-04T09:38:52.7340894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7341307Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7341713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:38:52.7342100Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:38:52.7342446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7342843Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7343008Z 2025-12-04T09:38:52.7343118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7343530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7343915Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7344330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7344704Z outputs = block( 2025-12-04T09:38:52.7345039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7345402Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7345785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7346160Z return func(*args, **kwargs) 2025-12-04T09:38:52.7346522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7346933Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7347327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:38:52.7347716Z hidden_states = self.act(hidden_states) 2025-12-04T09:38:52.7348062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:38:52.7348508Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:38:52.7348732Z 2025-12-04T09:38:52.7348841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7349241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7349681Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7350059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7350420Z outputs = block( 2025-12-04T09:38:52.7350727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7351083Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7351451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7351810Z return func(*args, **kwargs) 2025-12-04T09:38:52.7352170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7352565Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7352960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:38:52.7353336Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:38:52.7353686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7354070Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7354240Z 2025-12-04T09:38:52.7354347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7354744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7355129Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7355505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7355860Z outputs = block( 2025-12-04T09:38:52.7356172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7356521Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7356887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7357240Z return func(*args, **kwargs) 2025-12-04T09:38:52.7357601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7358019Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7358412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7358764Z return func(*args, **kwargs) 2025-12-04T09:38:52.7359122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:38:52.7359604Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:38:52.7360045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7360441Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7360627Z 2025-12-04T09:38:52.7360706Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7360934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7361327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7361709Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7362080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7362436Z outputs = block( 2025-12-04T09:38:52.7362734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7363076Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7363435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7363784Z return func(*args, **kwargs) 2025-12-04T09:38:52.7364146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7364536Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7364918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7365275Z return func(*args, **kwargs) 2025-12-04T09:38:52.7365636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:38:52.7366032Z attn_output, attn_weights = attention_interface( 2025-12-04T09:38:52.7366474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:38:52.7366959Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:52.7367153Z 2025-12-04T09:38:52.7367256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7367678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7368069Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7368463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7368834Z outputs = block( 2025-12-04T09:38:52.7369155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7369510Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7369888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7370265Z return func(*args, **kwargs) 2025-12-04T09:38:52.7370629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7371028Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7371554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7371954Z return func(*args, **kwargs) 2025-12-04T09:38:52.7372549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:38:52.7372978Z attn_output = self.c_proj(attn_output) 2025-12-04T09:38:52.7373340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7373738Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7373910Z 2025-12-04T09:38:52.7374012Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7374426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7374871Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7375256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7375627Z outputs = block( 2025-12-04T09:38:52.7375949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7376313Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7376684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7377055Z return func(*args, **kwargs) 2025-12-04T09:38:52.7377426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7377832Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7378236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:38:52.7378622Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:38:52.7378978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7379366Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7379543Z 2025-12-04T09:38:52.7379646Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7380055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7380446Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7380819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7381188Z outputs = block( 2025-12-04T09:38:52.7381507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7381856Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7382237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7382608Z return func(*args, **kwargs) 2025-12-04T09:38:52.7382975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7383391Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7383792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:38:52.7384180Z hidden_states = self.act(hidden_states) 2025-12-04T09:38:52.7384530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:38:52.7384982Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:38:52.7385220Z 2025-12-04T09:38:52.7385376Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7385820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7386216Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7386594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7386964Z outputs = block( 2025-12-04T09:38:52.7387282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7387637Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7388016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7388409Z return func(*args, **kwargs) 2025-12-04T09:38:52.7388782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7389183Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7389589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:38:52.7389982Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:38:52.7390334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7390727Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7390902Z 2025-12-04T09:38:52.7391007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7391419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7391809Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7392201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7392571Z outputs = block( 2025-12-04T09:38:52.7392890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7393243Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7393624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7393997Z return func(*args, **kwargs) 2025-12-04T09:38:52.7394360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:38:52.7394851Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:38:52.7395016Z 2025-12-04T09:38:52.7395122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7395555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7395966Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7396373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7396770Z outputs = block( 2025-12-04T09:38:52.7397085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7397467Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7397865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7398269Z return func(*args, **kwargs) 2025-12-04T09:38:52.7398648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7399068Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7399520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7399929Z return func(*args, **kwargs) 2025-12-04T09:38:52.7400311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:38:52.7400832Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:38:52.7401317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7401736Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7401914Z 2025-12-04T09:38:52.7402001Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7402227Z cudagraph partition due to non gpu ops 2025-12-04T09:38:52.7402504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7402943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7403372Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7403790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7404193Z outputs = block( 2025-12-04T09:38:52.7404531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7404918Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7405304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7405664Z return func(*args, **kwargs) 2025-12-04T09:38:52.7406028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7406417Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7406803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7407166Z return func(*args, **kwargs) 2025-12-04T09:38:52.7407531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:38:52.7407928Z attn_output, attn_weights = attention_interface( 2025-12-04T09:38:52.7408360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:38:52.7408846Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:38:52.7409036Z 2025-12-04T09:38:52.7409143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7409564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7409961Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7410353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7410732Z outputs = block( 2025-12-04T09:38:52.7411060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7411489Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7411881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7412261Z return func(*args, **kwargs) 2025-12-04T09:38:52.7412628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:38:52.7413042Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:38:52.7413560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7413948Z return func(*args, **kwargs) 2025-12-04T09:38:52.7414311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:38:52.7414708Z attn_output = self.c_proj(attn_output) 2025-12-04T09:38:52.7415057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7415440Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7415605Z 2025-12-04T09:38:52.7415705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7416106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7416506Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7416876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7417238Z outputs = block( 2025-12-04T09:38:52.7417552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7417905Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7418262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7418617Z return func(*args, **kwargs) 2025-12-04T09:38:52.7418974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7419380Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7419765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:38:52.7420144Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:38:52.7420492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7420870Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7421042Z 2025-12-04T09:38:52.7421146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7421543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7421926Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7422295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7422652Z outputs = block( 2025-12-04T09:38:52.7422965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7423306Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7423671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7424037Z return func(*args, **kwargs) 2025-12-04T09:38:52.7424383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7424763Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7425145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:38:52.7425566Z hidden_states = self.act(hidden_states) 2025-12-04T09:38:52.7425908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:38:52.7426341Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:38:52.7426573Z 2025-12-04T09:38:52.7426715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7427124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T09:38:52.7427529Z transformer_outputs = self.transformer( 2025-12-04T09:38:52.7427890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:38:52.7428242Z outputs = block( 2025-12-04T09:38:52.7428547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:38:52.7428882Z return super().__call__(*args, **kwargs) 2025-12-04T09:38:52.7429245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:38:52.7429616Z return func(*args, **kwargs) 2025-12-04T09:38:52.7429967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:38:52.7430346Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:38:52.7430731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:38:52.7431105Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:38:52.7431439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:38:52.7431819Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:38:52.7431984Z 2025-12-04T09:38:52.7432081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:38:52.7432467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1092, in forward 2025-12-04T09:38:52.7432860Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-12-04T09:38:52.7433023Z 2025-12-04T09:39:02.0471175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:02.0471785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:39:02.0472492Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:39:02.0472972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:39:02.0473449Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:39:02.0473698Z 2025-12-04T09:39:03.1020870Z Compilation time (from dynamo_timed): 16.139203539 2025-12-04T09:39:03.1206025Z pass 2025-12-04T09:39:03.1206438Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:03.1207484Z TIMING: gc:0.00345 entire_frame_compile:16.1392 _recursive_pre_grad_passes:0.00709 _recursive_joint_graph_passes:0.23131 _recursive_post_grad_passes:0.04627 async_compile.wait:1.4278 code_gen:9.55751 inductor_compile:10.24735 backend_compile:12.1825 total_wall_time:16.1392 2025-12-04T09:39:03.1208429Z STATS: call_* op count: 311 | FakeTensorMode.__torch_dispatch__:4780 | FakeTensor.__torch_dispatch__:2291 | ProxyTorchDispatchMode.__torch_dispatch__:916 2025-12-04T09:39:03.1208933Z Dynamo produced 2 graphs covering 311 ops with 2 graph breaks (1 unique) 2025-12-04T09:39:05.4358109Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:39:05.4359152Z import pynvml # type: ignore[import] 2025-12-04T09:39:08.8528845Z 2025-12-04T09:39:08.8545978Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T09:39:08.8547067Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T09:39:09.0578092Z 2025-12-04T09:39:09.0578975Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:39:09.0593774Z cpu eval ElectraForCausalLM 2025-12-04T09:39:09.2225215Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:09.3100251Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:09.3938083Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:17.0455850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0456304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0457592Z res = mod(**inputs) 2025-12-04T09:39:17.0458039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0458478Z outputs = self.electra( 2025-12-04T09:39:17.0458925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 787, in forward 2025-12-04T09:39:17.0459418Z hidden_states = self.embeddings_project(hidden_states) 2025-12-04T09:39:17.0459636Z 2025-12-04T09:39:17.0460911Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0461337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0461681Z res = mod(**inputs) 2025-12-04T09:39:17.0462237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0462669Z outputs = self.electra( 2025-12-04T09:39:17.0463068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0463475Z hidden_states = self.encoder( 2025-12-04T09:39:17.0463870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0464267Z layer_outputs = layer_module( 2025-12-04T09:39:17.0464622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0464993Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0465387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0465773Z return func(*args, **kwargs) 2025-12-04T09:39:17.0466158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0466559Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0466963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0467359Z return func(*args, **kwargs) 2025-12-04T09:39:17.0467756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0468157Z self_outputs = self.self( 2025-12-04T09:39:17.0468663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0469051Z return func(*args, **kwargs) 2025-12-04T09:39:17.0469424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0469842Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0470026Z 2025-12-04T09:39:17.0470150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0470778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0471160Z res = mod(**inputs) 2025-12-04T09:39:17.0471553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0471987Z outputs = self.electra( 2025-12-04T09:39:17.0472651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0473102Z hidden_states = self.encoder( 2025-12-04T09:39:17.0473497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0473894Z layer_outputs = layer_module( 2025-12-04T09:39:17.0474242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0474661Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0475047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0475421Z return func(*args, **kwargs) 2025-12-04T09:39:17.0475811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0476211Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0476587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0476948Z return func(*args, **kwargs) 2025-12-04T09:39:17.0477336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0477738Z self_outputs = self.self( 2025-12-04T09:39:17.0478097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0478472Z return func(*args, **kwargs) 2025-12-04T09:39:17.0478858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0479260Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0479393Z 2025-12-04T09:39:17.0479499Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0479868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0480193Z res = mod(**inputs) 2025-12-04T09:39:17.0480569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0480960Z outputs = self.electra( 2025-12-04T09:39:17.0481336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0481731Z hidden_states = self.encoder( 2025-12-04T09:39:17.0482115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0482510Z layer_outputs = layer_module( 2025-12-04T09:39:17.0482859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0483228Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0483604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0483983Z return func(*args, **kwargs) 2025-12-04T09:39:17.0484367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0484776Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0485157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0485599Z return func(*args, **kwargs) 2025-12-04T09:39:17.0486046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0486460Z self_outputs = self.self( 2025-12-04T09:39:17.0486849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0487252Z return func(*args, **kwargs) 2025-12-04T09:39:17.0487657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0488075Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0488224Z 2025-12-04T09:39:17.0488312Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0488570Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0488826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0489223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0489576Z res = mod(**inputs) 2025-12-04T09:39:17.0489985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0490407Z outputs = self.electra( 2025-12-04T09:39:17.0490836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0491268Z hidden_states = self.encoder( 2025-12-04T09:39:17.0491769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0492196Z layer_outputs = layer_module( 2025-12-04T09:39:17.0492578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0492980Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0493373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0493769Z return func(*args, **kwargs) 2025-12-04T09:39:17.0494169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0494591Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0494987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0495387Z return func(*args, **kwargs) 2025-12-04T09:39:17.0495789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0496259Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0496740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0497167Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0497313Z 2025-12-04T09:39:17.0497432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0497801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0498145Z res = mod(**inputs) 2025-12-04T09:39:17.0498538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0498951Z outputs = self.electra( 2025-12-04T09:39:17.0499350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0499741Z hidden_states = self.encoder( 2025-12-04T09:39:17.0500133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0500577Z layer_outputs = layer_module( 2025-12-04T09:39:17.0500989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0501354Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0501733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0502102Z return func(*args, **kwargs) 2025-12-04T09:39:17.0502472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0502871Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0503258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0503664Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0504096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0504579Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0505011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0505413Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0505552Z 2025-12-04T09:39:17.0505653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0506031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0506413Z res = mod(**inputs) 2025-12-04T09:39:17.0506775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0507158Z outputs = self.electra( 2025-12-04T09:39:17.0507520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0507901Z hidden_states = self.encoder( 2025-12-04T09:39:17.0508274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0508659Z layer_outputs = layer_module( 2025-12-04T09:39:17.0508988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0509341Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0509708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0510060Z return func(*args, **kwargs) 2025-12-04T09:39:17.0510429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0510823Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0511214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0511589Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0512004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0512465Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0512888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0513298Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0513671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0514008Z return self.act(input) 2025-12-04T09:39:17.0514115Z 2025-12-04T09:39:17.0514264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0514649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0514967Z res = mod(**inputs) 2025-12-04T09:39:17.0515331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0515706Z outputs = self.electra( 2025-12-04T09:39:17.0516071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0516453Z hidden_states = self.encoder( 2025-12-04T09:39:17.0516830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0517232Z layer_outputs = layer_module( 2025-12-04T09:39:17.0517580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0517941Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0518313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0518680Z return func(*args, **kwargs) 2025-12-04T09:39:17.0519053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0519447Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0519827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0520210Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0520625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0521100Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0521534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0521930Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0522062Z 2025-12-04T09:39:17.0522171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0522516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0522831Z res = mod(**inputs) 2025-12-04T09:39:17.0523194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0523581Z outputs = self.electra( 2025-12-04T09:39:17.0523950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0524330Z hidden_states = self.encoder( 2025-12-04T09:39:17.0524723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0525112Z layer_outputs = layer_module( 2025-12-04T09:39:17.0525450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0525815Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0526206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0526564Z return func(*args, **kwargs) 2025-12-04T09:39:17.0526933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0527326Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0527699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0528552Z return func(*args, **kwargs) 2025-12-04T09:39:17.0528963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0529346Z self_outputs = self.self( 2025-12-04T09:39:17.0529688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0530059Z return func(*args, **kwargs) 2025-12-04T09:39:17.0530434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0530834Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0530969Z 2025-12-04T09:39:17.0531071Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0531706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0532050Z res = mod(**inputs) 2025-12-04T09:39:17.0532430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0532842Z outputs = self.electra( 2025-12-04T09:39:17.0533326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0533792Z hidden_states = self.encoder( 2025-12-04T09:39:17.0534198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0534616Z layer_outputs = layer_module( 2025-12-04T09:39:17.0534986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0535375Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0535767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0536147Z return func(*args, **kwargs) 2025-12-04T09:39:17.0536533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0536931Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0537319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0537698Z return func(*args, **kwargs) 2025-12-04T09:39:17.0538090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0538473Z self_outputs = self.self( 2025-12-04T09:39:17.0538834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0539211Z return func(*args, **kwargs) 2025-12-04T09:39:17.0539585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0539989Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0540130Z 2025-12-04T09:39:17.0540235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0540594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0540909Z res = mod(**inputs) 2025-12-04T09:39:17.0541283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0541681Z outputs = self.electra( 2025-12-04T09:39:17.0542057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0542443Z hidden_states = self.encoder( 2025-12-04T09:39:17.0542878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0543268Z layer_outputs = layer_module( 2025-12-04T09:39:17.0543627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0543987Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0544371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0544739Z return func(*args, **kwargs) 2025-12-04T09:39:17.0545108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0545507Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0545885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0546270Z return func(*args, **kwargs) 2025-12-04T09:39:17.0546648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0547041Z self_outputs = self.self( 2025-12-04T09:39:17.0547401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0547764Z return func(*args, **kwargs) 2025-12-04T09:39:17.0548145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0548536Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0548666Z 2025-12-04T09:39:17.0548752Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0548951Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0549180Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0549527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0549884Z res = mod(**inputs) 2025-12-04T09:39:17.0550242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0550624Z outputs = self.electra( 2025-12-04T09:39:17.0550978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0551359Z hidden_states = self.encoder( 2025-12-04T09:39:17.0551730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0552113Z layer_outputs = layer_module( 2025-12-04T09:39:17.0552446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0552798Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0553173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0553536Z return func(*args, **kwargs) 2025-12-04T09:39:17.0553958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0554346Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0554712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0555065Z return func(*args, **kwargs) 2025-12-04T09:39:17.0555431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0555865Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0556293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0556680Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0556858Z 2025-12-04T09:39:17.0556977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0557323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0557635Z res = mod(**inputs) 2025-12-04T09:39:17.0557990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0558374Z outputs = self.electra( 2025-12-04T09:39:17.0558738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0559110Z hidden_states = self.encoder( 2025-12-04T09:39:17.0559482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0559885Z layer_outputs = layer_module( 2025-12-04T09:39:17.0560224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0560572Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0560942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0561309Z return func(*args, **kwargs) 2025-12-04T09:39:17.0561673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0562066Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0562455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0562837Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0563245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0563712Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0564146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0564535Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0564667Z 2025-12-04T09:39:17.0564769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0565124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0565442Z res = mod(**inputs) 2025-12-04T09:39:17.0565796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0566183Z outputs = self.electra( 2025-12-04T09:39:17.0566546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0566928Z hidden_states = self.encoder( 2025-12-04T09:39:17.0567300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0567682Z layer_outputs = layer_module( 2025-12-04T09:39:17.0568020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0568376Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0568740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0569104Z return func(*args, **kwargs) 2025-12-04T09:39:17.0569483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0569885Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0570335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0570750Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0571174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0571723Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0572196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0572949Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0573379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0573749Z return self.act(input) 2025-12-04T09:39:17.0573930Z 2025-12-04T09:39:17.0574036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0574395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0574722Z res = mod(**inputs) 2025-12-04T09:39:17.0575093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0575487Z outputs = self.electra( 2025-12-04T09:39:17.0575858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0576239Z hidden_states = self.encoder( 2025-12-04T09:39:17.0576623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0577014Z layer_outputs = layer_module( 2025-12-04T09:39:17.0577354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0577722Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0578110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0578482Z return func(*args, **kwargs) 2025-12-04T09:39:17.0578854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0579257Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0579659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0580047Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0580458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0580947Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0581399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0581798Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0581942Z 2025-12-04T09:39:17.0582048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0582403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0582719Z res = mod(**inputs) 2025-12-04T09:39:17.0583080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0583469Z outputs = self.electra( 2025-12-04T09:39:17.0583840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0584223Z hidden_states = self.encoder( 2025-12-04T09:39:17.0584606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0585053Z layer_outputs = layer_module( 2025-12-04T09:39:17.0585427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0585782Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0586167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0586548Z return func(*args, **kwargs) 2025-12-04T09:39:17.0586922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0587309Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0587684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0588095Z return func(*args, **kwargs) 2025-12-04T09:39:17.0588459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0588843Z self_outputs = self.self( 2025-12-04T09:39:17.0589195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0589554Z return func(*args, **kwargs) 2025-12-04T09:39:17.0589912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0590305Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0590436Z 2025-12-04T09:39:17.0590548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0590894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0591205Z res = mod(**inputs) 2025-12-04T09:39:17.0591573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0591959Z outputs = self.electra( 2025-12-04T09:39:17.0592316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0592698Z hidden_states = self.encoder( 2025-12-04T09:39:17.0593068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0593451Z layer_outputs = layer_module( 2025-12-04T09:39:17.0593781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0594133Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0594498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0594855Z return func(*args, **kwargs) 2025-12-04T09:39:17.0595267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0595657Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0596027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0596381Z return func(*args, **kwargs) 2025-12-04T09:39:17.0596750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0597129Z self_outputs = self.self( 2025-12-04T09:39:17.0597470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0597832Z return func(*args, **kwargs) 2025-12-04T09:39:17.0598200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0598618Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0598748Z 2025-12-04T09:39:17.0598867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0599218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0599531Z res = mod(**inputs) 2025-12-04T09:39:17.0599896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0600273Z outputs = self.electra( 2025-12-04T09:39:17.0600643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0601021Z hidden_states = self.encoder( 2025-12-04T09:39:17.0601390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0601790Z layer_outputs = layer_module( 2025-12-04T09:39:17.0602135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0602500Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0602868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0603236Z return func(*args, **kwargs) 2025-12-04T09:39:17.0603615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0604009Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0604392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0604762Z return func(*args, **kwargs) 2025-12-04T09:39:17.0605144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0605531Z self_outputs = self.self( 2025-12-04T09:39:17.0605895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0606268Z return func(*args, **kwargs) 2025-12-04T09:39:17.0606646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0607047Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0607186Z 2025-12-04T09:39:17.0607267Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0607490Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0607721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0608085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0608416Z res = mod(**inputs) 2025-12-04T09:39:17.0608798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0609194Z outputs = self.electra( 2025-12-04T09:39:17.0609575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0609977Z hidden_states = self.encoder( 2025-12-04T09:39:17.0610362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0610765Z layer_outputs = layer_module( 2025-12-04T09:39:17.0611119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0611581Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0611978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0612373Z return func(*args, **kwargs) 2025-12-04T09:39:17.0612810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0613234Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0613618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0613991Z return func(*args, **kwargs) 2025-12-04T09:39:17.0614367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0614807Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0615270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0615712Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0615854Z 2025-12-04T09:39:17.0615971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0616346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0616671Z res = mod(**inputs) 2025-12-04T09:39:17.0617040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0617422Z outputs = self.electra( 2025-12-04T09:39:17.0617795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0618182Z hidden_states = self.encoder( 2025-12-04T09:39:17.0618562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0618950Z layer_outputs = layer_module( 2025-12-04T09:39:17.0619292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0619658Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0620026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0620402Z return func(*args, **kwargs) 2025-12-04T09:39:17.0620780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0621185Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0621576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0621973Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0622401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0622877Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0623312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0623716Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0623851Z 2025-12-04T09:39:17.0623962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0624317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0624638Z res = mod(**inputs) 2025-12-04T09:39:17.0625003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0625393Z outputs = self.electra( 2025-12-04T09:39:17.0625756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0626149Z hidden_states = self.encoder( 2025-12-04T09:39:17.0626576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0626981Z layer_outputs = layer_module( 2025-12-04T09:39:17.0627318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0627679Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0628053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0628416Z return func(*args, **kwargs) 2025-12-04T09:39:17.0628795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0629194Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0629590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0629994Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0630424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0630899Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0631329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0631741Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0632126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0632471Z return self.act(input) 2025-12-04T09:39:17.0632581Z 2025-12-04T09:39:17.0632926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0633290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0633617Z res = mod(**inputs) 2025-12-04T09:39:17.0633993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0634385Z outputs = self.electra( 2025-12-04T09:39:17.0634764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0635161Z hidden_states = self.encoder( 2025-12-04T09:39:17.0635597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0635991Z layer_outputs = layer_module( 2025-12-04T09:39:17.0636339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0636703Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0637082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0637448Z return func(*args, **kwargs) 2025-12-04T09:39:17.0637824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0638223Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0638611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0638993Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0639409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0639887Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0640383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0640778Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0640945Z 2025-12-04T09:39:17.0641073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0641419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0641744Z res = mod(**inputs) 2025-12-04T09:39:17.0642113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0642501Z outputs = self.electra( 2025-12-04T09:39:17.0642867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0643251Z hidden_states = self.encoder( 2025-12-04T09:39:17.0643630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0644019Z layer_outputs = layer_module( 2025-12-04T09:39:17.0644360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0644726Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0645096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0645451Z return func(*args, **kwargs) 2025-12-04T09:39:17.0645818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0646210Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0646574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0646937Z return func(*args, **kwargs) 2025-12-04T09:39:17.0647307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0647685Z self_outputs = self.self( 2025-12-04T09:39:17.0648044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0648417Z return func(*args, **kwargs) 2025-12-04T09:39:17.0648791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0649191Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0649328Z 2025-12-04T09:39:17.0649432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0649788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0650109Z res = mod(**inputs) 2025-12-04T09:39:17.0650468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0650870Z outputs = self.electra( 2025-12-04T09:39:17.0651266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0651760Z hidden_states = self.encoder( 2025-12-04T09:39:17.0652161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0652574Z layer_outputs = layer_module( 2025-12-04T09:39:17.0652936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0653314Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0653687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0654059Z return func(*args, **kwargs) 2025-12-04T09:39:17.0654443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0654880Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0655283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0655714Z return func(*args, **kwargs) 2025-12-04T09:39:17.0656095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0656479Z self_outputs = self.self( 2025-12-04T09:39:17.0656839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0657211Z return func(*args, **kwargs) 2025-12-04T09:39:17.0657582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0658005Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0658147Z 2025-12-04T09:39:17.0658252Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0658615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0658934Z res = mod(**inputs) 2025-12-04T09:39:17.0659304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0659706Z outputs = self.electra( 2025-12-04T09:39:17.0660071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0660464Z hidden_states = self.encoder( 2025-12-04T09:39:17.0660848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0661237Z layer_outputs = layer_module( 2025-12-04T09:39:17.0661579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0661946Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0662328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0662698Z return func(*args, **kwargs) 2025-12-04T09:39:17.0663073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0663475Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0663860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0664230Z return func(*args, **kwargs) 2025-12-04T09:39:17.0664593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0664969Z self_outputs = self.self( 2025-12-04T09:39:17.0665317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0665666Z return func(*args, **kwargs) 2025-12-04T09:39:17.0666032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0666425Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0666555Z 2025-12-04T09:39:17.0666633Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0666844Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0667074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0667420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0667725Z res = mod(**inputs) 2025-12-04T09:39:17.0668088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0668475Z outputs = self.electra( 2025-12-04T09:39:17.0668875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0669278Z hidden_states = self.encoder( 2025-12-04T09:39:17.0669651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0670033Z layer_outputs = layer_module( 2025-12-04T09:39:17.0670371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0670740Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0671105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0671462Z return func(*args, **kwargs) 2025-12-04T09:39:17.0671857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0672451Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0672855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0673221Z return func(*args, **kwargs) 2025-12-04T09:39:17.0673605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0674055Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0674501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0674900Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0675048Z 2025-12-04T09:39:17.0675151Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0675521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0675838Z res = mod(**inputs) 2025-12-04T09:39:17.0676197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0676585Z outputs = self.electra( 2025-12-04T09:39:17.0676951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0677336Z hidden_states = self.encoder( 2025-12-04T09:39:17.0677728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0678117Z layer_outputs = layer_module( 2025-12-04T09:39:17.0678464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0678823Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0679202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0679586Z return func(*args, **kwargs) 2025-12-04T09:39:17.0679949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0680345Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0680736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0681124Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0681532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0681996Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0682426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0682888Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0683048Z 2025-12-04T09:39:17.0683149Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0683499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0683812Z res = mod(**inputs) 2025-12-04T09:39:17.0684160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0684542Z outputs = self.electra( 2025-12-04T09:39:17.0684915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0685297Z hidden_states = self.encoder( 2025-12-04T09:39:17.0685661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0686070Z layer_outputs = layer_module( 2025-12-04T09:39:17.0686410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0686759Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0687137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0687517Z return func(*args, **kwargs) 2025-12-04T09:39:17.0687885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0688272Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0688660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0689040Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0689460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0689915Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0690342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0690768Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0691139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0691580Z return self.act(input) 2025-12-04T09:39:17.0691710Z 2025-12-04T09:39:17.0691821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0692203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0692543Z res = mod(**inputs) 2025-12-04T09:39:17.0692919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0693317Z outputs = self.electra( 2025-12-04T09:39:17.0693766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0694154Z hidden_states = self.encoder( 2025-12-04T09:39:17.0694539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0694928Z layer_outputs = layer_module( 2025-12-04T09:39:17.0695263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0695624Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0695996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0696366Z return func(*args, **kwargs) 2025-12-04T09:39:17.0696779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0697190Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0697581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0697953Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0698364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0698843Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0699265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0699660Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0699797Z 2025-12-04T09:39:17.0699895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0700232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0700537Z res = mod(**inputs) 2025-12-04T09:39:17.0700878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0701246Z outputs = self.electra( 2025-12-04T09:39:17.0701595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0701956Z hidden_states = self.encoder( 2025-12-04T09:39:17.0702318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0702684Z layer_outputs = layer_module( 2025-12-04T09:39:17.0703010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0703344Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0703701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0704054Z return func(*args, **kwargs) 2025-12-04T09:39:17.0704408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0704779Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0705137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0705488Z return func(*args, **kwargs) 2025-12-04T09:39:17.0705835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0706201Z self_outputs = self.self( 2025-12-04T09:39:17.0706546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0706894Z return func(*args, **kwargs) 2025-12-04T09:39:17.0707245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0707621Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0707748Z 2025-12-04T09:39:17.0707852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0708181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0708479Z res = mod(**inputs) 2025-12-04T09:39:17.0708827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0709198Z outputs = self.electra( 2025-12-04T09:39:17.0709543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0709950Z hidden_states = self.encoder( 2025-12-04T09:39:17.0710333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0710702Z layer_outputs = layer_module( 2025-12-04T09:39:17.0711021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0711365Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0711722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0712064Z return func(*args, **kwargs) 2025-12-04T09:39:17.0712424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0712829Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0713194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0713544Z return func(*args, **kwargs) 2025-12-04T09:39:17.0713919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0714293Z self_outputs = self.self( 2025-12-04T09:39:17.0714636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0714991Z return func(*args, **kwargs) 2025-12-04T09:39:17.0715358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0715741Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0715869Z 2025-12-04T09:39:17.0715972Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0716320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0716632Z res = mod(**inputs) 2025-12-04T09:39:17.0716984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0717363Z outputs = self.electra( 2025-12-04T09:39:17.0717722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0718099Z hidden_states = self.encoder( 2025-12-04T09:39:17.0718461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0718838Z layer_outputs = layer_module( 2025-12-04T09:39:17.0719172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0719521Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0719877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0720233Z return func(*args, **kwargs) 2025-12-04T09:39:17.0720598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0720974Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0721337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0721693Z return func(*args, **kwargs) 2025-12-04T09:39:17.0722054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0722421Z self_outputs = self.self( 2025-12-04T09:39:17.0722766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0723123Z return func(*args, **kwargs) 2025-12-04T09:39:17.0723511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0723934Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0724067Z 2025-12-04T09:39:17.0724144Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0724346Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0724562Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0724901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0725205Z res = mod(**inputs) 2025-12-04T09:39:17.0725545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0725940Z outputs = self.electra( 2025-12-04T09:39:17.0726304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0726677Z hidden_states = self.encoder( 2025-12-04T09:39:17.0727033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0727404Z layer_outputs = layer_module( 2025-12-04T09:39:17.0727728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0728072Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0728423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0728773Z return func(*args, **kwargs) 2025-12-04T09:39:17.0729137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0729519Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0729901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0730273Z return func(*args, **kwargs) 2025-12-04T09:39:17.0730639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0731068Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0731572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0731974Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0732109Z 2025-12-04T09:39:17.0732218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0732560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0732885Z res = mod(**inputs) 2025-12-04T09:39:17.0733261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0733649Z outputs = self.electra( 2025-12-04T09:39:17.0734026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0734414Z hidden_states = self.encoder( 2025-12-04T09:39:17.0734792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0735246Z layer_outputs = layer_module( 2025-12-04T09:39:17.0735577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0735933Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0736289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0736651Z return func(*args, **kwargs) 2025-12-04T09:39:17.0737051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0737462Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0737841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0738219Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0738634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0739098Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0739512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0739922Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0740053Z 2025-12-04T09:39:17.0740166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0740509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0740825Z res = mod(**inputs) 2025-12-04T09:39:17.0741184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0741562Z outputs = self.electra( 2025-12-04T09:39:17.0741912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0742278Z hidden_states = self.encoder( 2025-12-04T09:39:17.0742631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0742993Z layer_outputs = layer_module( 2025-12-04T09:39:17.0743304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0743641Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0743992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0744329Z return func(*args, **kwargs) 2025-12-04T09:39:17.0744682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0745056Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0745423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0745782Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0746180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0746620Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0747017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0747416Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0747771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0748086Z return self.act(input) 2025-12-04T09:39:17.0748189Z 2025-12-04T09:39:17.0748286Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0748616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0748914Z res = mod(**inputs) 2025-12-04T09:39:17.0749255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0749609Z outputs = self.electra( 2025-12-04T09:39:17.0749986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0750361Z hidden_states = self.encoder( 2025-12-04T09:39:17.0750707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0751070Z layer_outputs = layer_module( 2025-12-04T09:39:17.0751388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0751721Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0752062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0752403Z return func(*args, **kwargs) 2025-12-04T09:39:17.0752773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0753143Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0753511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0753868Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0754255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0754691Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0755102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0755475Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0755601Z 2025-12-04T09:39:17.0755704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0756035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0756352Z res = mod(**inputs) 2025-12-04T09:39:17.0756692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0757045Z outputs = self.electra( 2025-12-04T09:39:17.0757388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0757759Z hidden_states = self.encoder( 2025-12-04T09:39:17.0758128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0758483Z layer_outputs = layer_module( 2025-12-04T09:39:17.0758808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0759162Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0759516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0759864Z return func(*args, **kwargs) 2025-12-04T09:39:17.0760222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0760607Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0760985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0761326Z return func(*args, **kwargs) 2025-12-04T09:39:17.0761676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0762036Z self_outputs = self.self( 2025-12-04T09:39:17.0762364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0762706Z return func(*args, **kwargs) 2025-12-04T09:39:17.0763095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0763497Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0763631Z 2025-12-04T09:39:17.0763727Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0764068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0764375Z res = mod(**inputs) 2025-12-04T09:39:17.0764721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0765095Z outputs = self.electra( 2025-12-04T09:39:17.0765451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0765848Z hidden_states = self.encoder( 2025-12-04T09:39:17.0766213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0766586Z layer_outputs = layer_module( 2025-12-04T09:39:17.0766915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0767254Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0767615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0767967Z return func(*args, **kwargs) 2025-12-04T09:39:17.0768327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0768702Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0769071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0769434Z return func(*args, **kwargs) 2025-12-04T09:39:17.0769800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0770180Z self_outputs = self.self( 2025-12-04T09:39:17.0770532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0770890Z return func(*args, **kwargs) 2025-12-04T09:39:17.0771252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0771780Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0771928Z 2025-12-04T09:39:17.0772051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0772576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0772923Z res = mod(**inputs) 2025-12-04T09:39:17.0773306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0773704Z outputs = self.electra( 2025-12-04T09:39:17.0774075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0774460Z hidden_states = self.encoder( 2025-12-04T09:39:17.0774826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0775198Z layer_outputs = layer_module( 2025-12-04T09:39:17.0775522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0775870Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0776233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0776582Z return func(*args, **kwargs) 2025-12-04T09:39:17.0777049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0777463Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0777828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0778175Z return func(*args, **kwargs) 2025-12-04T09:39:17.0778539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0778942Z self_outputs = self.self( 2025-12-04T09:39:17.0779286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0779658Z return func(*args, **kwargs) 2025-12-04T09:39:17.0780027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0780407Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0780532Z 2025-12-04T09:39:17.0780609Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0780814Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0781039Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0781379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0781677Z res = mod(**inputs) 2025-12-04T09:39:17.0782027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0782399Z outputs = self.electra( 2025-12-04T09:39:17.0782745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0783117Z hidden_states = self.encoder( 2025-12-04T09:39:17.0783488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0783858Z layer_outputs = layer_module( 2025-12-04T09:39:17.0784177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0784524Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0784877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0785221Z return func(*args, **kwargs) 2025-12-04T09:39:17.0785581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0785962Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0786321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0786668Z return func(*args, **kwargs) 2025-12-04T09:39:17.0787025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0787456Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0787884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0788268Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0788404Z 2025-12-04T09:39:17.0788505Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0788850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0789153Z res = mod(**inputs) 2025-12-04T09:39:17.0789514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0789891Z outputs = self.electra( 2025-12-04T09:39:17.0790291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0790688Z hidden_states = self.encoder( 2025-12-04T09:39:17.0791068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0791459Z layer_outputs = layer_module( 2025-12-04T09:39:17.0791791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0792151Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0792524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0792910Z return func(*args, **kwargs) 2025-12-04T09:39:17.0793281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0793674Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0794071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0794452Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0794861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0795324Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0795752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0796136Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0796279Z 2025-12-04T09:39:17.0796379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0796732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0797047Z res = mod(**inputs) 2025-12-04T09:39:17.0797398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0797813Z outputs = self.electra( 2025-12-04T09:39:17.0798177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0798559Z hidden_states = self.encoder( 2025-12-04T09:39:17.0798924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0799306Z layer_outputs = layer_module( 2025-12-04T09:39:17.0799645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0799996Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0800350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0800698Z return func(*args, **kwargs) 2025-12-04T09:39:17.0801052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0801420Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0801791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0802154Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0802553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0803002Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0803465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0803906Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0804251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0804568Z return self.act(input) 2025-12-04T09:39:17.0804674Z 2025-12-04T09:39:17.0804769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0805097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0805387Z res = mod(**inputs) 2025-12-04T09:39:17.0805725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0806088Z outputs = self.electra( 2025-12-04T09:39:17.0806459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0806834Z hidden_states = self.encoder( 2025-12-04T09:39:17.0807204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0807577Z layer_outputs = layer_module( 2025-12-04T09:39:17.0807899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0808249Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0808611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0808961Z return func(*args, **kwargs) 2025-12-04T09:39:17.0809324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0809712Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0810095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0810462Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0810870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0811398Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0811881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0812314Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0812468Z 2025-12-04T09:39:17.0812582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0812969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0813298Z res = mod(**inputs) 2025-12-04T09:39:17.0813681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0814067Z outputs = self.electra( 2025-12-04T09:39:17.0814429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0814802Z hidden_states = self.encoder( 2025-12-04T09:39:17.0815176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0815550Z layer_outputs = layer_module( 2025-12-04T09:39:17.0815882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0816226Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0816594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0816959Z return func(*args, **kwargs) 2025-12-04T09:39:17.0817354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0817758Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0818126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0818483Z return func(*args, **kwargs) 2025-12-04T09:39:17.0818839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0819214Z self_outputs = self.self( 2025-12-04T09:39:17.0819555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0819899Z return func(*args, **kwargs) 2025-12-04T09:39:17.0820274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0820659Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0820789Z 2025-12-04T09:39:17.0820894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0821228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0821537Z res = mod(**inputs) 2025-12-04T09:39:17.0821887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0822257Z outputs = self.electra( 2025-12-04T09:39:17.0822647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0823017Z hidden_states = self.encoder( 2025-12-04T09:39:17.0823373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0823733Z layer_outputs = layer_module( 2025-12-04T09:39:17.0824059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0824405Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0824763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0825173Z return func(*args, **kwargs) 2025-12-04T09:39:17.0825532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0825916Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0826305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0826659Z return func(*args, **kwargs) 2025-12-04T09:39:17.0827019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0827388Z self_outputs = self.self( 2025-12-04T09:39:17.0827722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0828076Z return func(*args, **kwargs) 2025-12-04T09:39:17.0828434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0828815Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0828940Z 2025-12-04T09:39:17.0829039Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0829372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0829677Z res = mod(**inputs) 2025-12-04T09:39:17.0830019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0830398Z outputs = self.electra( 2025-12-04T09:39:17.0830790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0831178Z hidden_states = self.encoder( 2025-12-04T09:39:17.0831532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0831905Z layer_outputs = layer_module( 2025-12-04T09:39:17.0832229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0832563Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0832922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0833294Z return func(*args, **kwargs) 2025-12-04T09:39:17.0833653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0834028Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0834390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0834741Z return func(*args, **kwargs) 2025-12-04T09:39:17.0835098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0835463Z self_outputs = self.self( 2025-12-04T09:39:17.0835806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0836157Z return func(*args, **kwargs) 2025-12-04T09:39:17.0836507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0836888Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0837020Z 2025-12-04T09:39:17.0837098Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0837306Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0837523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0837861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0838165Z res = mod(**inputs) 2025-12-04T09:39:17.0838504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0838878Z outputs = self.electra( 2025-12-04T09:39:17.0839129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0839196Z hidden_states = self.encoder( 2025-12-04T09:39:17.0839444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0839520Z layer_outputs = layer_module( 2025-12-04T09:39:17.0839731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0839807Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0840042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0840106Z return func(*args, **kwargs) 2025-12-04T09:39:17.0840360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0840436Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0840718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0840792Z return func(*args, **kwargs) 2025-12-04T09:39:17.0841071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0841211Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0841469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0841548Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0841552Z 2025-12-04T09:39:17.0841656Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0841846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0841906Z res = mod(**inputs) 2025-12-04T09:39:17.0842166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0842249Z outputs = self.electra( 2025-12-04T09:39:17.0842509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0842577Z hidden_states = self.encoder( 2025-12-04T09:39:17.0842821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0842895Z layer_outputs = layer_module( 2025-12-04T09:39:17.0843104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0843176Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0843412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0843475Z return func(*args, **kwargs) 2025-12-04T09:39:17.0843726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0843806Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0844050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0844132Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0844413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0844532Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0844775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0844850Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0844854Z 2025-12-04T09:39:17.0844958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0845155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0845215Z res = mod(**inputs) 2025-12-04T09:39:17.0845465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0845528Z outputs = self.electra( 2025-12-04T09:39:17.0845776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0845840Z hidden_states = self.encoder( 2025-12-04T09:39:17.0846077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0846149Z layer_outputs = layer_module( 2025-12-04T09:39:17.0846352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0846430Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0846653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0846748Z return func(*args, **kwargs) 2025-12-04T09:39:17.0847013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0847091Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0847334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0847410Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0847687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0847805Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0848047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0848171Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0848384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0848452Z return self.act(input) 2025-12-04T09:39:17.0848455Z 2025-12-04T09:39:17.0848561Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0848744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0848802Z res = mod(**inputs) 2025-12-04T09:39:17.0849055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0849117Z outputs = self.electra( 2025-12-04T09:39:17.0849359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0849435Z hidden_states = self.encoder( 2025-12-04T09:39:17.0849681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0849756Z layer_outputs = layer_module( 2025-12-04T09:39:17.0849962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0850034Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0850268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0850332Z return func(*args, **kwargs) 2025-12-04T09:39:17.0850573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0850655Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0850897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0850977Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0851256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0851452Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0851716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0851794Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0851798Z 2025-12-04T09:39:17.0851906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0852094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0852155Z res = mod(**inputs) 2025-12-04T09:39:17.0852446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0852518Z outputs = self.electra( 2025-12-04T09:39:17.0852835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0852945Z hidden_states = self.encoder( 2025-12-04T09:39:17.0853206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0853282Z layer_outputs = layer_module( 2025-12-04T09:39:17.0853516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0853589Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0853842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0853908Z return func(*args, **kwargs) 2025-12-04T09:39:17.0854205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0854286Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0854529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0854600Z return func(*args, **kwargs) 2025-12-04T09:39:17.0854859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0854926Z self_outputs = self.self( 2025-12-04T09:39:17.0855177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0855241Z return func(*args, **kwargs) 2025-12-04T09:39:17.0855508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0855589Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0855592Z 2025-12-04T09:39:17.0855696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0855903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0855964Z res = mod(**inputs) 2025-12-04T09:39:17.0856235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0856301Z outputs = self.electra( 2025-12-04T09:39:17.0856563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0856636Z hidden_states = self.encoder( 2025-12-04T09:39:17.0856894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0856963Z layer_outputs = layer_module( 2025-12-04T09:39:17.0857196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0857273Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0857527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0857595Z return func(*args, **kwargs) 2025-12-04T09:39:17.0857857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0857945Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0858193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0858260Z return func(*args, **kwargs) 2025-12-04T09:39:17.0858538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0858609Z self_outputs = self.self( 2025-12-04T09:39:17.0858949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0859030Z return func(*args, **kwargs) 2025-12-04T09:39:17.0859277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0859360Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0859363Z 2025-12-04T09:39:17.0859461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0859657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0859717Z res = mod(**inputs) 2025-12-04T09:39:17.0859968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0860055Z outputs = self.electra( 2025-12-04T09:39:17.0860304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0860370Z hidden_states = self.encoder( 2025-12-04T09:39:17.0860622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0860686Z layer_outputs = layer_module( 2025-12-04T09:39:17.0860900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0860972Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0861205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0861277Z return func(*args, **kwargs) 2025-12-04T09:39:17.0861522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0861599Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0861837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0861905Z return func(*args, **kwargs) 2025-12-04T09:39:17.0862156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0862222Z self_outputs = self.self( 2025-12-04T09:39:17.0862455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0862528Z return func(*args, **kwargs) 2025-12-04T09:39:17.0862780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0862865Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0862869Z 2025-12-04T09:39:17.0862948Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0863025Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0863134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0863329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0863390Z res = mod(**inputs) 2025-12-04T09:39:17.0863658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0863723Z outputs = self.electra( 2025-12-04T09:39:17.0863985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0864050Z hidden_states = self.encoder( 2025-12-04T09:39:17.0864309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0864383Z layer_outputs = layer_module( 2025-12-04T09:39:17.0864633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0864724Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0864967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0865032Z return func(*args, **kwargs) 2025-12-04T09:39:17.0865290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0865365Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0865598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0865670Z return func(*args, **kwargs) 2025-12-04T09:39:17.0865922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0866078Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0866340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0866418Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0866422Z 2025-12-04T09:39:17.0866528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0866715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0866774Z res = mod(**inputs) 2025-12-04T09:39:17.0867033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0867096Z outputs = self.electra( 2025-12-04T09:39:17.0867348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0867415Z hidden_states = self.encoder( 2025-12-04T09:39:17.0867665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0867740Z layer_outputs = layer_module( 2025-12-04T09:39:17.0867949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0868020Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0868257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0868322Z return func(*args, **kwargs) 2025-12-04T09:39:17.0868575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0868654Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0868900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0868981Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0869263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0869383Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0869628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0869705Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0869708Z 2025-12-04T09:39:17.0869814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0870000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0870065Z res = mod(**inputs) 2025-12-04T09:39:17.0870317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0870421Z outputs = self.electra( 2025-12-04T09:39:17.0870699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0870764Z hidden_states = self.encoder( 2025-12-04T09:39:17.0871006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0871080Z layer_outputs = layer_module( 2025-12-04T09:39:17.0871286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0871363Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0871590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0871671Z return func(*args, **kwargs) 2025-12-04T09:39:17.0871932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0872010Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0872448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0872538Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0872824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0872946Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0873192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0873298Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0873513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0873582Z return self.act(input) 2025-12-04T09:39:17.0873587Z 2025-12-04T09:39:17.0873696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0873882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0873943Z res = mod(**inputs) 2025-12-04T09:39:17.0874202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0874266Z outputs = self.electra( 2025-12-04T09:39:17.0874510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0874588Z hidden_states = self.encoder( 2025-12-04T09:39:17.0874833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0874920Z layer_outputs = layer_module( 2025-12-04T09:39:17.0875129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0875203Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0875442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0875505Z return func(*args, **kwargs) 2025-12-04T09:39:17.0875754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0875830Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0876071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0876150Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0876494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0876655Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0876910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0876986Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0876989Z 2025-12-04T09:39:17.0877095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0877281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0877341Z res = mod(**inputs) 2025-12-04T09:39:17.0877598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0877687Z outputs = self.electra( 2025-12-04T09:39:17.0877945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0878011Z hidden_states = self.encoder( 2025-12-04T09:39:17.0878259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0878333Z layer_outputs = layer_module( 2025-12-04T09:39:17.0878542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0878614Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0878857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0878920Z return func(*args, **kwargs) 2025-12-04T09:39:17.0879174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0879252Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0879484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0879556Z return func(*args, **kwargs) 2025-12-04T09:39:17.0879803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0879868Z self_outputs = self.self( 2025-12-04T09:39:17.0880106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0880167Z return func(*args, **kwargs) 2025-12-04T09:39:17.0880420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0880495Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0880500Z 2025-12-04T09:39:17.0880596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0880791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0880851Z res = mod(**inputs) 2025-12-04T09:39:17.0881109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0881171Z outputs = self.electra( 2025-12-04T09:39:17.0881417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0881489Z hidden_states = self.encoder( 2025-12-04T09:39:17.0881734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0881797Z layer_outputs = layer_module( 2025-12-04T09:39:17.0882016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0882089Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0882356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0882458Z return func(*args, **kwargs) 2025-12-04T09:39:17.0882714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0882797Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0883035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0883107Z return func(*args, **kwargs) 2025-12-04T09:39:17.0883361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0883425Z self_outputs = self.self( 2025-12-04T09:39:17.0883674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0883740Z return func(*args, **kwargs) 2025-12-04T09:39:17.0883983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0884064Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0884068Z 2025-12-04T09:39:17.0884165Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0884358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0884417Z res = mod(**inputs) 2025-12-04T09:39:17.0884672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0884744Z outputs = self.electra( 2025-12-04T09:39:17.0884994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0885063Z hidden_states = self.encoder( 2025-12-04T09:39:17.0885321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0885389Z layer_outputs = layer_module( 2025-12-04T09:39:17.0885609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0885684Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0885921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0885992Z return func(*args, **kwargs) 2025-12-04T09:39:17.0886236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0886319Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0886554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0886621Z return func(*args, **kwargs) 2025-12-04T09:39:17.0886878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0886943Z self_outputs = self.self( 2025-12-04T09:39:17.0887177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0887250Z return func(*args, **kwargs) 2025-12-04T09:39:17.0887498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0887583Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0887587Z 2025-12-04T09:39:17.0887664Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0887742Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0887848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0888070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0888148Z res = mod(**inputs) 2025-12-04T09:39:17.0888410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0888475Z outputs = self.electra( 2025-12-04T09:39:17.0888731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0888798Z hidden_states = self.encoder( 2025-12-04T09:39:17.0889045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0889121Z layer_outputs = layer_module( 2025-12-04T09:39:17.0889335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0889428Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0889673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0889741Z return func(*args, **kwargs) 2025-12-04T09:39:17.0889997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0890073Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0910200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0910381Z return func(*args, **kwargs) 2025-12-04T09:39:17.0910698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0910840Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0911117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0911208Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0911216Z 2025-12-04T09:39:17.0911333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0911538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0911604Z res = mod(**inputs) 2025-12-04T09:39:17.0911876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0911948Z outputs = self.electra( 2025-12-04T09:39:17.0912199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0912275Z hidden_states = self.encoder( 2025-12-04T09:39:17.0912526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0912604Z layer_outputs = layer_module( 2025-12-04T09:39:17.0912825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0912907Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0913154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0913224Z return func(*args, **kwargs) 2025-12-04T09:39:17.0913475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0913568Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0913813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0913895Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0914296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0914447Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0914708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0914787Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0914791Z 2025-12-04T09:39:17.0914904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0915098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0915160Z res = mod(**inputs) 2025-12-04T09:39:17.0915426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0915522Z outputs = self.electra( 2025-12-04T09:39:17.0915782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0915857Z hidden_states = self.encoder( 2025-12-04T09:39:17.0916108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0916182Z layer_outputs = layer_module( 2025-12-04T09:39:17.0916403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0916482Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0916725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0916791Z return func(*args, **kwargs) 2025-12-04T09:39:17.0917047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0917127Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0917374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0917453Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0917736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0917851Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0918112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0918218Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0918432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0918499Z return self.act(input) 2025-12-04T09:39:17.0918502Z 2025-12-04T09:39:17.0918606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0918802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0918861Z res = mod(**inputs) 2025-12-04T09:39:17.0919121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0919186Z outputs = self.electra( 2025-12-04T09:39:17.0919433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0919507Z hidden_states = self.encoder( 2025-12-04T09:39:17.0919752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0919832Z layer_outputs = layer_module( 2025-12-04T09:39:17.0920048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0920157Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0920407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0920474Z return func(*args, **kwargs) 2025-12-04T09:39:17.0920717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0920803Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0921047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0921117Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0921402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0921549Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0921805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0921884Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0921888Z 2025-12-04T09:39:17.0921988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0922187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0922246Z res = mod(**inputs) 2025-12-04T09:39:17.0922502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0922566Z outputs = self.electra( 2025-12-04T09:39:17.0922807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0922883Z hidden_states = self.encoder( 2025-12-04T09:39:17.0923130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0923197Z layer_outputs = layer_module( 2025-12-04T09:39:17.0923412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0923487Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0923730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0923795Z return func(*args, **kwargs) 2025-12-04T09:39:17.0924041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0924128Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0924351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0924426Z return func(*args, **kwargs) 2025-12-04T09:39:17.0924665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0924734Z self_outputs = self.self( 2025-12-04T09:39:17.0924960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0925022Z return func(*args, **kwargs) 2025-12-04T09:39:17.0925258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0925341Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0925345Z 2025-12-04T09:39:17.0925441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0925630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0925688Z res = mod(**inputs) 2025-12-04T09:39:17.0925963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0926050Z outputs = self.electra( 2025-12-04T09:39:17.0926290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0926362Z hidden_states = self.encoder( 2025-12-04T09:39:17.0926600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0926666Z layer_outputs = layer_module( 2025-12-04T09:39:17.0926880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0926954Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0927199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0927277Z return func(*args, **kwargs) 2025-12-04T09:39:17.0927524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0927607Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0927832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0927895Z return func(*args, **kwargs) 2025-12-04T09:39:17.0928148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0928215Z self_outputs = self.self( 2025-12-04T09:39:17.0928442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0928515Z return func(*args, **kwargs) 2025-12-04T09:39:17.0928764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0928847Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0928851Z 2025-12-04T09:39:17.0928949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0929134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0929202Z res = mod(**inputs) 2025-12-04T09:39:17.0929448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0929519Z outputs = self.electra( 2025-12-04T09:39:17.0929765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0929833Z hidden_states = self.encoder( 2025-12-04T09:39:17.0930083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0930152Z layer_outputs = layer_module( 2025-12-04T09:39:17.0930364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0930444Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0930670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0930741Z return func(*args, **kwargs) 2025-12-04T09:39:17.0930984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0931062Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0931419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0931509Z return func(*args, **kwargs) 2025-12-04T09:39:17.0931867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0931972Z self_outputs = self.self( 2025-12-04T09:39:17.0932237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0932318Z return func(*args, **kwargs) 2025-12-04T09:39:17.0932604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0932680Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0932683Z 2025-12-04T09:39:17.0932771Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0932846Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0932953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0933167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0933226Z res = mod(**inputs) 2025-12-04T09:39:17.0933486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0933551Z outputs = self.electra( 2025-12-04T09:39:17.0933794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0933868Z hidden_states = self.encoder( 2025-12-04T09:39:17.0934112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0934185Z layer_outputs = layer_module( 2025-12-04T09:39:17.0934404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0934475Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0934708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0934771Z return func(*args, **kwargs) 2025-12-04T09:39:17.0935014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0935099Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0935323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0935389Z return func(*args, **kwargs) 2025-12-04T09:39:17.0935646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0935770Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0936021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0936100Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0936106Z 2025-12-04T09:39:17.0936204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0936397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0936456Z res = mod(**inputs) 2025-12-04T09:39:17.0936739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0936803Z outputs = self.electra( 2025-12-04T09:39:17.0937045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0937115Z hidden_states = self.encoder( 2025-12-04T09:39:17.0937356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0937421Z layer_outputs = layer_module( 2025-12-04T09:39:17.0937668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0937756Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0937990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0938052Z return func(*args, **kwargs) 2025-12-04T09:39:17.0938294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0938377Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0938618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0938689Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0938990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0939104Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0939354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0939430Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0939434Z 2025-12-04T09:39:17.0939528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0939720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0939779Z res = mod(**inputs) 2025-12-04T09:39:17.0940035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0940098Z outputs = self.electra( 2025-12-04T09:39:17.0940343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0940416Z hidden_states = self.encoder( 2025-12-04T09:39:17.0940663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0940730Z layer_outputs = layer_module( 2025-12-04T09:39:17.0940945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0941017Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0941253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0941315Z return func(*args, **kwargs) 2025-12-04T09:39:17.0941558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0941645Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0941888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0941970Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0942246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0942358Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0942609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0942715Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0942918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0942991Z return self.act(input) 2025-12-04T09:39:17.0942996Z 2025-12-04T09:39:17.0943093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0943319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0943396Z res = mod(**inputs) 2025-12-04T09:39:17.0943648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0943720Z outputs = self.electra( 2025-12-04T09:39:17.0943965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0944038Z hidden_states = self.encoder( 2025-12-04T09:39:17.0944281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0944346Z layer_outputs = layer_module( 2025-12-04T09:39:17.0944562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0944656Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0944888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0944961Z return func(*args, **kwargs) 2025-12-04T09:39:17.0945205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0945288Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0945528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0945598Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0945878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0946003Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0946257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0946333Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0946336Z 2025-12-04T09:39:17.0946430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0946619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0946679Z res = mod(**inputs) 2025-12-04T09:39:17.0946927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0946998Z outputs = self.electra( 2025-12-04T09:39:17.0947241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0947313Z hidden_states = self.encoder( 2025-12-04T09:39:17.0947558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0947624Z layer_outputs = layer_module( 2025-12-04T09:39:17.0947839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0947922Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0948142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0948212Z return func(*args, **kwargs) 2025-12-04T09:39:17.0948447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0948529Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0948750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0948815Z return func(*args, **kwargs) 2025-12-04T09:39:17.0949097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0949180Z self_outputs = self.self( 2025-12-04T09:39:17.0949412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0949474Z return func(*args, **kwargs) 2025-12-04T09:39:17.0949720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0949800Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0949803Z 2025-12-04T09:39:17.0949898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0950082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0950162Z res = mod(**inputs) 2025-12-04T09:39:17.0950406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0950476Z outputs = self.electra( 2025-12-04T09:39:17.0950713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0950774Z hidden_states = self.encoder( 2025-12-04T09:39:17.0951016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0951079Z layer_outputs = layer_module( 2025-12-04T09:39:17.0951278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0951353Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0951571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0951639Z return func(*args, **kwargs) 2025-12-04T09:39:17.0951878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0951952Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0952176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0952236Z return func(*args, **kwargs) 2025-12-04T09:39:17.0952482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0952545Z self_outputs = self.self( 2025-12-04T09:39:17.0952762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0952829Z return func(*args, **kwargs) 2025-12-04T09:39:17.0953066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0953137Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0953150Z 2025-12-04T09:39:17.0953246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0953425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0953488Z res = mod(**inputs) 2025-12-04T09:39:17.0953729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0953790Z outputs = self.electra( 2025-12-04T09:39:17.0954036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0954098Z hidden_states = self.encoder( 2025-12-04T09:39:17.0954345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0954410Z layer_outputs = layer_module( 2025-12-04T09:39:17.0954935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0955041Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0955267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0955329Z return func(*args, **kwargs) 2025-12-04T09:39:17.0955575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0955650Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0955879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0955940Z return func(*args, **kwargs) 2025-12-04T09:39:17.0956195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0956272Z self_outputs = self.self( 2025-12-04T09:39:17.0956492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0956557Z return func(*args, **kwargs) 2025-12-04T09:39:17.0956803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0956878Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0956881Z 2025-12-04T09:39:17.0956966Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0957041Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0957137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0957332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0957392Z res = mod(**inputs) 2025-12-04T09:39:17.0957642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0957713Z outputs = self.electra( 2025-12-04T09:39:17.0957957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0958040Z hidden_states = self.encoder( 2025-12-04T09:39:17.0958276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0958339Z layer_outputs = layer_module( 2025-12-04T09:39:17.0958551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0958621Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0958850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0958915Z return func(*args, **kwargs) 2025-12-04T09:39:17.0959155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0959239Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0959459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0959522Z return func(*args, **kwargs) 2025-12-04T09:39:17.0959769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0959885Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0960130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0960207Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0960211Z 2025-12-04T09:39:17.0960305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0960523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0960596Z res = mod(**inputs) 2025-12-04T09:39:17.0960853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0960916Z outputs = self.electra( 2025-12-04T09:39:17.0961161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0961233Z hidden_states = self.encoder( 2025-12-04T09:39:17.0961478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0961542Z layer_outputs = layer_module( 2025-12-04T09:39:17.0961770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0961843Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0962075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0962138Z return func(*args, **kwargs) 2025-12-04T09:39:17.0962374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0962457Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0962700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0962774Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0963044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0963157Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0963405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0963480Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0963483Z 2025-12-04T09:39:17.0963576Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0963761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0963821Z res = mod(**inputs) 2025-12-04T09:39:17.0964068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0964129Z outputs = self.electra( 2025-12-04T09:39:17.0964366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0964439Z hidden_states = self.encoder( 2025-12-04T09:39:17.0964678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0964750Z layer_outputs = layer_module( 2025-12-04T09:39:17.0964953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0965023Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0965247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0965309Z return func(*args, **kwargs) 2025-12-04T09:39:17.0965543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0965624Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0965857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0965934Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0966231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0966358Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0966602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0966705Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0966902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0966972Z return self.act(input) 2025-12-04T09:39:17.0966975Z 2025-12-04T09:39:17.0967068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0967255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0967334Z res = mod(**inputs) 2025-12-04T09:39:17.0967580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0967651Z outputs = self.electra( 2025-12-04T09:39:17.0967891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0967961Z hidden_states = self.encoder( 2025-12-04T09:39:17.0968200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0968264Z layer_outputs = layer_module( 2025-12-04T09:39:17.0968474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0968544Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0968769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0968841Z return func(*args, **kwargs) 2025-12-04T09:39:17.0969079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0969162Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0969397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0969463Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0969741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0969863Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0970115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0970190Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0970193Z 2025-12-04T09:39:17.0970289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0970480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0970537Z res = mod(**inputs) 2025-12-04T09:39:17.0970780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0970851Z outputs = self.electra( 2025-12-04T09:39:17.0971096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0971175Z hidden_states = self.encoder( 2025-12-04T09:39:17.0971537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0971620Z layer_outputs = layer_module( 2025-12-04T09:39:17.0971902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0972007Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0972491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0972567Z return func(*args, **kwargs) 2025-12-04T09:39:17.0972820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0972908Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0973145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0973212Z return func(*args, **kwargs) 2025-12-04T09:39:17.0973476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0973606Z self_outputs = self.self( 2025-12-04T09:39:17.0973866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0973930Z return func(*args, **kwargs) 2025-12-04T09:39:17.0974181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T09:39:17.0974265Z query_layer = self.query(hidden_states) 2025-12-04T09:39:17.0974269Z 2025-12-04T09:39:17.0974367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0974562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0974631Z res = mod(**inputs) 2025-12-04T09:39:17.0974894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0974970Z outputs = self.electra( 2025-12-04T09:39:17.0975232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0975300Z hidden_states = self.encoder( 2025-12-04T09:39:17.0975568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0975634Z layer_outputs = layer_module( 2025-12-04T09:39:17.0975862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0975937Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0976177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0976249Z return func(*args, **kwargs) 2025-12-04T09:39:17.0976507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0976588Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0976833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0976898Z return func(*args, **kwargs) 2025-12-04T09:39:17.0977165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0977230Z self_outputs = self.self( 2025-12-04T09:39:17.0977470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0977541Z return func(*args, **kwargs) 2025-12-04T09:39:17.0977797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T09:39:17.0977873Z key_layer = self.key(current_states) 2025-12-04T09:39:17.0977884Z 2025-12-04T09:39:17.0977982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0978243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0978339Z res = mod(**inputs) 2025-12-04T09:39:17.0978595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0978668Z outputs = self.electra( 2025-12-04T09:39:17.0978921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0978989Z hidden_states = self.encoder( 2025-12-04T09:39:17.0979245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0979311Z layer_outputs = layer_module( 2025-12-04T09:39:17.0979542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0979627Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0979858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0979930Z return func(*args, **kwargs) 2025-12-04T09:39:17.0980178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0980255Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0980496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0980560Z return func(*args, **kwargs) 2025-12-04T09:39:17.0980815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T09:39:17.0980882Z self_outputs = self.self( 2025-12-04T09:39:17.0981116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0981192Z return func(*args, **kwargs) 2025-12-04T09:39:17.0981440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T09:39:17.0981515Z value_layer = self.value(current_states) 2025-12-04T09:39:17.0981525Z 2025-12-04T09:39:17.0981602Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0981678Z cudagraph partition due to non gpu ops 2025-12-04T09:39:17.0981781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0981969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0982030Z res = mod(**inputs) 2025-12-04T09:39:17.0982290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0982357Z outputs = self.electra( 2025-12-04T09:39:17.0982609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0982682Z hidden_states = self.encoder( 2025-12-04T09:39:17.0982930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0983001Z layer_outputs = layer_module( 2025-12-04T09:39:17.0983214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0983288Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0983526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0983590Z return func(*args, **kwargs) 2025-12-04T09:39:17.0983849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T09:39:17.0983958Z self_attention_outputs = self.attention( 2025-12-04T09:39:17.0984208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0984278Z return func(*args, **kwargs) 2025-12-04T09:39:17.0984527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T09:39:17.0984649Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:39:17.0984904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T09:39:17.0984981Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0984985Z 2025-12-04T09:39:17.0985089Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0985298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0985362Z res = mod(**inputs) 2025-12-04T09:39:17.0985627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0985691Z outputs = self.electra( 2025-12-04T09:39:17.0985948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0986016Z hidden_states = self.encoder( 2025-12-04T09:39:17.0986265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0986339Z layer_outputs = layer_module( 2025-12-04T09:39:17.0986551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0986624Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0986865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0986931Z return func(*args, **kwargs) 2025-12-04T09:39:17.0987187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0987267Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0987514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0987596Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0987880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0988007Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0988259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T09:39:17.0988337Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0988342Z 2025-12-04T09:39:17.0988446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0988631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0988691Z res = mod(**inputs) 2025-12-04T09:39:17.0988952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0989016Z outputs = self.electra( 2025-12-04T09:39:17.0989269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0989337Z hidden_states = self.encoder( 2025-12-04T09:39:17.0989587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0989664Z layer_outputs = layer_module( 2025-12-04T09:39:17.0989908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0989999Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0990241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0990306Z return func(*args, **kwargs) 2025-12-04T09:39:17.0990569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0990645Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0990886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0990982Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0991261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T09:39:17.0991377Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:39:17.0991624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T09:39:17.0991728Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:39:17.0991940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:39:17.0992005Z return self.act(input) 2025-12-04T09:39:17.0992008Z 2025-12-04T09:39:17.0992104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0992302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0992364Z res = mod(**inputs) 2025-12-04T09:39:17.0992629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T09:39:17.0992696Z outputs = self.electra( 2025-12-04T09:39:17.0992956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T09:39:17.0993029Z hidden_states = self.encoder( 2025-12-04T09:39:17.0993273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T09:39:17.0993343Z layer_outputs = layer_module( 2025-12-04T09:39:17.0993552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:17.0993622Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:17.0993856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:17.0993921Z return func(*args, **kwargs) 2025-12-04T09:39:17.0994168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T09:39:17.0994257Z layer_output = apply_chunking_to_forward( 2025-12-04T09:39:17.0994496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:39:17.0994572Z return forward_fn(*input_tensors) 2025-12-04T09:39:17.0994847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T09:39:17.0994970Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:39:17.0995224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T09:39:17.0995299Z hidden_states = self.dense(hidden_states) 2025-12-04T09:39:17.0995304Z 2025-12-04T09:39:17.0995409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0995623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0995706Z res = mod(**inputs) 2025-12-04T09:39:17.0995961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-12-04T09:39:17.0996131Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-12-04T09:39:17.0996377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 630, in forward 2025-12-04T09:39:17.0996479Z hidden_states = self.dense(generator_hidden_states) 2025-12-04T09:39:17.0996482Z 2025-12-04T09:39:17.0996577Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0996766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0996842Z res = mod(**inputs) 2025-12-04T09:39:17.0997094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-12-04T09:39:17.0997270Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-12-04T09:39:17.0997274Z 2025-12-04T09:39:17.0997368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:17.0997557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:17.0997615Z res = mod(**inputs) 2025-12-04T09:39:17.0997863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1554, in forward 2025-12-04T09:39:17.0997936Z lm_loss = self.loss_function( 2025-12-04T09:39:17.0998164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:39:17.0998334Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:39:17.0998575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:39:17.0998759Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:39:17.0998762Z 2025-12-04T09:39:26.8171784Z Compilation time (from dynamo_timed): 16.881770467 2025-12-04T09:39:26.8245723Z pass 2025-12-04T09:39:26.8246374Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:26.8247413Z TIMING: _recursive_pre_grad_passes:0.00814 _recursive_joint_graph_passes:0.43733 _recursive_post_grad_passes:0.06247 async_compile.wait:0.78244 code_gen:9.32951 inductor_compile:10.57422 backend_compile:13.89908 gc:0.00046 entire_frame_compile:16.88177 total_wall_time:16.88177 2025-12-04T09:39:26.8248479Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:8666 | FakeTensor.__torch_dispatch__:4346 | ProxyTorchDispatchMode.__torch_dispatch__:2555 2025-12-04T09:39:26.8249023Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-12-04T09:39:29.1949161Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:39:29.1950743Z import pynvml # type: ignore[import] 2025-12-04T09:39:32.5533227Z 2025-12-04T09:39:33.8008901Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:39:33.8009316Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:39:33.8027869Z cpu eval GPT2ForSequenceClassification 2025-12-04T09:39:34.7338399Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:35.0968709Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:35.4536910Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:42.1917336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1917822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1918147Z res = mod(**inputs) 2025-12-04T09:39:42.1918537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1918998Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1919396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:39:42.1919831Z causal_mask = create_causal_mask( 2025-12-04T09:39:42.1920581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:39:42.1921093Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:39:42.1921612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:39:42.1922100Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:39:42.1922574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-12-04T09:39:42.1923085Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-12-04T09:39:42.1923345Z 2025-12-04T09:39:42.1923427Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1923665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1924023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1924336Z res = mod(**inputs) 2025-12-04T09:39:42.1924714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1394, in forward 2025-12-04T09:39:42.1925145Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-12-04T09:39:42.1925314Z 2025-12-04T09:39:42.1925422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1925765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1926081Z res = mod(**inputs) 2025-12-04T09:39:42.1926452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1926850Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1927234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.1927616Z outputs = block( 2025-12-04T09:39:42.1927948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1928313Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1928723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1929122Z return func(*args, **kwargs) 2025-12-04T09:39:42.1929518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.1929932Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.1930356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1930760Z return func(*args, **kwargs) 2025-12-04T09:39:42.1931150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.1931979Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.1932537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.1932964Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.1933136Z 2025-12-04T09:39:42.1933245Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1933613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1933939Z res = mod(**inputs) 2025-12-04T09:39:42.1934311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1934708Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1935132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:39:42.1935527Z causal_mask = create_causal_mask( 2025-12-04T09:39:42.1935902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:39:42.1936409Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:39:42.1936945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:39:42.1937390Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:39:42.1937824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-12-04T09:39:42.1938311Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-12-04T09:39:42.1938529Z 2025-12-04T09:39:42.1938727Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1938971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1939333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1939669Z res = mod(**inputs) 2025-12-04T09:39:42.1940034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1940429Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1940809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:39:42.1941196Z causal_mask = create_causal_mask( 2025-12-04T09:39:42.1941575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:39:42.1942103Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:39:42.1942660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:39:42.1943141Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:39:42.1943635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T09:39:42.1944070Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T09:39:42.1944240Z 2025-12-04T09:39:42.1944346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1944705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1945030Z res = mod(**inputs) 2025-12-04T09:39:42.1945382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1945782Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1946219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T09:39:42.1946677Z causal_mask = create_causal_mask( 2025-12-04T09:39:42.1947071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T09:39:42.1947597Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T09:39:42.1948137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T09:39:42.1948578Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T09:39:42.1949032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T09:39:42.1949487Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T09:39:42.1949648Z 2025-12-04T09:39:42.1949738Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1949972Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1950331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1950654Z res = mod(**inputs) 2025-12-04T09:39:42.1951014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1951407Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1951793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.1952163Z outputs = block( 2025-12-04T09:39:42.1952483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1952850Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1953233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1953605Z return func(*args, **kwargs) 2025-12-04T09:39:42.1953967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.1954360Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.1954748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1955114Z return func(*args, **kwargs) 2025-12-04T09:39:42.1955502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.1955933Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.1956407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.1956916Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.1957128Z 2025-12-04T09:39:42.1957231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1957591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1957931Z res = mod(**inputs) 2025-12-04T09:39:42.1958304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1958722Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1959133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.1959520Z outputs = block( 2025-12-04T09:39:42.1959871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1960255Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1960691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1961101Z return func(*args, **kwargs) 2025-12-04T09:39:42.1961494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.1961918Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.1962321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1962719Z return func(*args, **kwargs) 2025-12-04T09:39:42.1963105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.1963523Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.1963923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.1964363Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.1964560Z 2025-12-04T09:39:42.1964677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1965069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1965411Z res = mod(**inputs) 2025-12-04T09:39:42.1965796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1966222Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1966632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.1967033Z outputs = block( 2025-12-04T09:39:42.1967383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1967821Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1968228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1968661Z return func(*args, **kwargs) 2025-12-04T09:39:42.1969062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.1969506Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.1969939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.1970359Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.1970746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.1971168Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.1971450Z 2025-12-04T09:39:42.1971570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1971962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1972493Z res = mod(**inputs) 2025-12-04T09:39:42.1972879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1973296Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1973706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.1974094Z outputs = block( 2025-12-04T09:39:42.1974439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1974823Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1975226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1975708Z return func(*args, **kwargs) 2025-12-04T09:39:42.1976106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.1976587Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.1977028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.1977437Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.1977836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.1978327Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.1978574Z 2025-12-04T09:39:42.1978686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1979118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1979472Z res = mod(**inputs) 2025-12-04T09:39:42.1979861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1980283Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1980699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.1981101Z outputs = block( 2025-12-04T09:39:42.1981446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1981832Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1982239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1982643Z return func(*args, **kwargs) 2025-12-04T09:39:42.1983040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.1983483Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.1983917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.1984342Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.1984724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.1985154Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.1985339Z 2025-12-04T09:39:42.1985459Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1985837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1986185Z res = mod(**inputs) 2025-12-04T09:39:42.1986568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1986985Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1987395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.1987797Z outputs = block( 2025-12-04T09:39:42.1988143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1988536Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1988934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1989332Z return func(*args, **kwargs) 2025-12-04T09:39:42.1989725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.1990148Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.1990597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1991010Z return func(*args, **kwargs) 2025-12-04T09:39:42.1991403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.1991927Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.1992425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.1992853Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.1993036Z 2025-12-04T09:39:42.1993136Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.1993389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.1993795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.1994152Z res = mod(**inputs) 2025-12-04T09:39:42.1994525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.1994946Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.1995357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.1995750Z outputs = block( 2025-12-04T09:39:42.1996084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.1996468Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.1996858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1997227Z return func(*args, **kwargs) 2025-12-04T09:39:42.1997593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.1997992Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.1998380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.1998743Z return func(*args, **kwargs) 2025-12-04T09:39:42.1999110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.1999512Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.1999973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2000482Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2000683Z 2025-12-04T09:39:42.2000793Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2001177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2001510Z res = mod(**inputs) 2025-12-04T09:39:42.2001890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2002306Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2002716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2003098Z outputs = block( 2025-12-04T09:39:42.2003436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2003820Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2004212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2004611Z return func(*args, **kwargs) 2025-12-04T09:39:42.2005016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2005433Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2005813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2006185Z return func(*args, **kwargs) 2025-12-04T09:39:42.2006554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2006945Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2007296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2007711Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2007917Z 2025-12-04T09:39:42.2008038Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2008409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2008738Z res = mod(**inputs) 2025-12-04T09:39:42.2009099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2009496Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2009891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2010285Z outputs = block( 2025-12-04T09:39:42.2010629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2011012Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2011516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2011946Z return func(*args, **kwargs) 2025-12-04T09:39:42.2012370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2012805Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2013248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2013690Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2014084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2014517Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2014714Z 2025-12-04T09:39:42.2014828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2015233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2015578Z res = mod(**inputs) 2025-12-04T09:39:42.2015981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2016425Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2016859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2017271Z outputs = block( 2025-12-04T09:39:42.2017626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2018032Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2018445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2018864Z return func(*args, **kwargs) 2025-12-04T09:39:42.2019279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2019741Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2020273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2020676Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2021025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2021475Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2021703Z 2025-12-04T09:39:42.2021803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2022153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2022465Z res = mod(**inputs) 2025-12-04T09:39:42.2022828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2023222Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2023661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2024033Z outputs = block( 2025-12-04T09:39:42.2024348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2024703Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2025076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2025442Z return func(*args, **kwargs) 2025-12-04T09:39:42.2025800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2026212Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2026618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2027005Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2027369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2027764Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2027933Z 2025-12-04T09:39:42.2028045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2028393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2028712Z res = mod(**inputs) 2025-12-04T09:39:42.2029067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2029453Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2029845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2030215Z outputs = block( 2025-12-04T09:39:42.2030536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2030888Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2031264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2031637Z return func(*args, **kwargs) 2025-12-04T09:39:42.2032007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2032393Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2032778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2033147Z return func(*args, **kwargs) 2025-12-04T09:39:42.2033577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2034098Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2034536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2034912Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2035071Z 2025-12-04T09:39:42.2035151Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2035378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2035715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2036014Z res = mod(**inputs) 2025-12-04T09:39:42.2036354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2036750Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2037119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2037466Z outputs = block( 2025-12-04T09:39:42.2037771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2038116Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2038471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2038818Z return func(*args, **kwargs) 2025-12-04T09:39:42.2039164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2039542Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2039900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2040253Z return func(*args, **kwargs) 2025-12-04T09:39:42.2040599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2040978Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2041389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2041844Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2042017Z 2025-12-04T09:39:42.2042119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2042455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2042753Z res = mod(**inputs) 2025-12-04T09:39:42.2043091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2043467Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2043831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2044173Z outputs = block( 2025-12-04T09:39:42.2044467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2044802Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2045153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2045504Z return func(*args, **kwargs) 2025-12-04T09:39:42.2045853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2046226Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2046625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2046998Z return func(*args, **kwargs) 2025-12-04T09:39:42.2047352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2047716Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2048054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2048435Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2048595Z 2025-12-04T09:39:42.2048715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2049049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2049373Z res = mod(**inputs) 2025-12-04T09:39:42.2049717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2050088Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2050468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2050825Z outputs = block( 2025-12-04T09:39:42.2051145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2051606Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2052035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2052453Z return func(*args, **kwargs) 2025-12-04T09:39:42.2052837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2053280Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2053682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2054065Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2054407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2054798Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2054965Z 2025-12-04T09:39:42.2055073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2055420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2055723Z res = mod(**inputs) 2025-12-04T09:39:42.2056070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2056455Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2056824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2057200Z outputs = block( 2025-12-04T09:39:42.2057503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2057851Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2058210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2058573Z return func(*args, **kwargs) 2025-12-04T09:39:42.2058934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2059380Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2059781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2060165Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2060546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2061008Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2061234Z 2025-12-04T09:39:42.2061332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2061676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2061982Z res = mod(**inputs) 2025-12-04T09:39:42.2062316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2062696Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2063065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2063428Z outputs = block( 2025-12-04T09:39:42.2063737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2064091Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2064450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2064809Z return func(*args, **kwargs) 2025-12-04T09:39:42.2065157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2065543Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2065919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2066300Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2066654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2067040Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2067206Z 2025-12-04T09:39:42.2067305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2067654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2067964Z res = mod(**inputs) 2025-12-04T09:39:42.2068308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2068679Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2069052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2069414Z outputs = block( 2025-12-04T09:39:42.2069720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2070075Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2070445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2070809Z return func(*args, **kwargs) 2025-12-04T09:39:42.2071160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:39:42.2071572Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:39:42.2071725Z 2025-12-04T09:39:42.2071833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2072181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2072609Z res = mod(**inputs) 2025-12-04T09:39:42.2072966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2073354Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2073801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2074186Z outputs = block( 2025-12-04T09:39:42.2074498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2074849Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2075210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2075570Z return func(*args, **kwargs) 2025-12-04T09:39:42.2075931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2076309Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2076719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2077084Z return func(*args, **kwargs) 2025-12-04T09:39:42.2077444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2077921Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2078376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2078764Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2078933Z 2025-12-04T09:39:42.2079020Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2079247Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2079598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2079910Z res = mod(**inputs) 2025-12-04T09:39:42.2080252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2080643Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2081024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2081385Z outputs = block( 2025-12-04T09:39:42.2081697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2082031Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2082380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2082717Z return func(*args, **kwargs) 2025-12-04T09:39:42.2083069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2083449Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2083817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2084166Z return func(*args, **kwargs) 2025-12-04T09:39:42.2084518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2084905Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2085333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2085783Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2085960Z 2025-12-04T09:39:42.2086059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2086401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2086704Z res = mod(**inputs) 2025-12-04T09:39:42.2087089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2087478Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2087842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2088182Z outputs = block( 2025-12-04T09:39:42.2088482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2088821Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2089171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2089520Z return func(*args, **kwargs) 2025-12-04T09:39:42.2089890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2090264Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2090623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2090974Z return func(*args, **kwargs) 2025-12-04T09:39:42.2091373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2091779Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2092151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2092572Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2092750Z 2025-12-04T09:39:42.2092866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2093227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2093599Z res = mod(**inputs) 2025-12-04T09:39:42.2093940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2094317Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2094678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2095029Z outputs = block( 2025-12-04T09:39:42.2095335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2095671Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2096030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2096386Z return func(*args, **kwargs) 2025-12-04T09:39:42.2096740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2097129Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2097525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2097904Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2098255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2098635Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2098806Z 2025-12-04T09:39:42.2098910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2099258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2099562Z res = mod(**inputs) 2025-12-04T09:39:42.2099912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2100330Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2100713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2101086Z outputs = block( 2025-12-04T09:39:42.2101400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2101750Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2102108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2102472Z return func(*args, **kwargs) 2025-12-04T09:39:42.2102828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2103245Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2103638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2104022Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2104356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2104785Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2105002Z 2025-12-04T09:39:42.2105101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2105440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2105745Z res = mod(**inputs) 2025-12-04T09:39:42.2106077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2106451Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2106820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2107172Z outputs = block( 2025-12-04T09:39:42.2107468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2107813Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2108170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2108521Z return func(*args, **kwargs) 2025-12-04T09:39:42.2108866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2109253Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2109635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2110003Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2110345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2110725Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2110887Z 2025-12-04T09:39:42.2110990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2111321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2111622Z res = mod(**inputs) 2025-12-04T09:39:42.2111958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2112331Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2112690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2113042Z outputs = block( 2025-12-04T09:39:42.2113380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2113736Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2114097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2114454Z return func(*args, **kwargs) 2025-12-04T09:39:42.2114804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2115171Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2115543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2115899Z return func(*args, **kwargs) 2025-12-04T09:39:42.2116247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2116740Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2117179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2117553Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2117716Z 2025-12-04T09:39:42.2117794Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2118022Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2118363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2118668Z res = mod(**inputs) 2025-12-04T09:39:42.2118996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2119373Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2119738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2120080Z outputs = block( 2025-12-04T09:39:42.2120400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2120742Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2121107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2121452Z return func(*args, **kwargs) 2025-12-04T09:39:42.2121806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2122181Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2122543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2122891Z return func(*args, **kwargs) 2025-12-04T09:39:42.2123283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2123677Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2124106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2124562Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2124748Z 2025-12-04T09:39:42.2124848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2125191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2125501Z res = mod(**inputs) 2025-12-04T09:39:42.2125856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2126238Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2126639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2127083Z outputs = block( 2025-12-04T09:39:42.2127395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2127758Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2128128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2128497Z return func(*args, **kwargs) 2025-12-04T09:39:42.2128865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2129255Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2129635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2130025Z return func(*args, **kwargs) 2025-12-04T09:39:42.2130395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2130777Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2131133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2131613Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2131790Z 2025-12-04T09:39:42.2131907Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2132263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2132595Z res = mod(**inputs) 2025-12-04T09:39:42.2132957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2133356Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2133741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2134109Z outputs = block( 2025-12-04T09:39:42.2134428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2134775Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2135147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2135525Z return func(*args, **kwargs) 2025-12-04T09:39:42.2135880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2136274Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2136673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2137050Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2137393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2137788Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2137959Z 2025-12-04T09:39:42.2138060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2138410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2138717Z res = mod(**inputs) 2025-12-04T09:39:42.2139064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2139459Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2139824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2140177Z outputs = block( 2025-12-04T09:39:42.2140529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2140891Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2141240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2141601Z return func(*args, **kwargs) 2025-12-04T09:39:42.2141955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2142343Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2142718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2143083Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2143457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2143885Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2144116Z 2025-12-04T09:39:42.2144214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2144558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2144864Z res = mod(**inputs) 2025-12-04T09:39:42.2145195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2145568Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2145936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2146290Z outputs = block( 2025-12-04T09:39:42.2146591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2146941Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2147303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2147654Z return func(*args, **kwargs) 2025-12-04T09:39:42.2148020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2148419Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2148816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2149194Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2149542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2149927Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2150089Z 2025-12-04T09:39:42.2150198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2150530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2150832Z res = mod(**inputs) 2025-12-04T09:39:42.2151171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2151536Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2151909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2152260Z outputs = block( 2025-12-04T09:39:42.2152567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2152911Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2153308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2153674Z return func(*args, **kwargs) 2025-12-04T09:39:42.2154020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:39:42.2154417Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:39:42.2154578Z 2025-12-04T09:39:42.2154677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2155021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2155324Z res = mod(**inputs) 2025-12-04T09:39:42.2155667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2156047Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2156423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2156787Z outputs = block( 2025-12-04T09:39:42.2157110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2157451Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2157804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2158156Z return func(*args, **kwargs) 2025-12-04T09:39:42.2158504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2158891Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2159246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2159595Z return func(*args, **kwargs) 2025-12-04T09:39:42.2159943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2160415Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2160861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2161248Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2161411Z 2025-12-04T09:39:42.2161497Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2161719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2162068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2162380Z res = mod(**inputs) 2025-12-04T09:39:42.2162721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2163107Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2163485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2163849Z outputs = block( 2025-12-04T09:39:42.2164153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2164500Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2164869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2165240Z return func(*args, **kwargs) 2025-12-04T09:39:42.2165604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2166001Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2166389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2166791Z return func(*args, **kwargs) 2025-12-04T09:39:42.2167180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2167582Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2168185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2168683Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2168879Z 2025-12-04T09:39:42.2168987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2169369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2169706Z res = mod(**inputs) 2025-12-04T09:39:42.2170107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2170529Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2170937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2171396Z outputs = block( 2025-12-04T09:39:42.2171763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2172168Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2172766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2173158Z return func(*args, **kwargs) 2025-12-04T09:39:42.2173534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2173945Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2174345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2174708Z return func(*args, **kwargs) 2025-12-04T09:39:42.2175076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2175500Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2175866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2176262Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2176439Z 2025-12-04T09:39:42.2176543Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2176894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2177202Z res = mod(**inputs) 2025-12-04T09:39:42.2177559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2177947Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2178319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2178689Z outputs = block( 2025-12-04T09:39:42.2179006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2179359Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2179719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2180078Z return func(*args, **kwargs) 2025-12-04T09:39:42.2180438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2180845Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2181329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2181749Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2182107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2182504Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2182683Z 2025-12-04T09:39:42.2182786Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2183158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2183481Z res = mod(**inputs) 2025-12-04T09:39:42.2183832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2184261Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2184649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2185016Z outputs = block( 2025-12-04T09:39:42.2185338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2185700Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2186080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2186452Z return func(*args, **kwargs) 2025-12-04T09:39:42.2186824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2187247Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2187649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2188037Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2188392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2188850Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2189080Z 2025-12-04T09:39:42.2189186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2189542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2189868Z res = mod(**inputs) 2025-12-04T09:39:42.2190228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2190626Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2191016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2191391Z outputs = block( 2025-12-04T09:39:42.2191712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2192082Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2192468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2192852Z return func(*args, **kwargs) 2025-12-04T09:39:42.2193275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2193694Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2194103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2194501Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2194866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2195317Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2195510Z 2025-12-04T09:39:42.2195624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2195985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2196310Z res = mod(**inputs) 2025-12-04T09:39:42.2196676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2197079Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2197473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2197856Z outputs = block( 2025-12-04T09:39:42.2198184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2198568Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2198953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2199328Z return func(*args, **kwargs) 2025-12-04T09:39:42.2199702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2200094Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2200487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2200852Z return func(*args, **kwargs) 2025-12-04T09:39:42.2201222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2201721Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2202187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2202596Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2202765Z 2025-12-04T09:39:42.2202852Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2203084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2203452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2203773Z res = mod(**inputs) 2025-12-04T09:39:42.2204123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2204518Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2204905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2205277Z outputs = block( 2025-12-04T09:39:42.2205597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2205969Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2206372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2206757Z return func(*args, **kwargs) 2025-12-04T09:39:42.2207154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2207577Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2207984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2208369Z return func(*args, **kwargs) 2025-12-04T09:39:42.2208765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2209244Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2209733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2210267Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2210468Z 2025-12-04T09:39:42.2210576Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2210961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2211365Z res = mod(**inputs) 2025-12-04T09:39:42.2211767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2212212Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2212686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2213083Z outputs = block( 2025-12-04T09:39:42.2213444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2213843Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2214242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2214687Z return func(*args, **kwargs) 2025-12-04T09:39:42.2215062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2215467Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2215855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2216237Z return func(*args, **kwargs) 2025-12-04T09:39:42.2216639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2217077Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2217468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2217892Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2218071Z 2025-12-04T09:39:42.2218190Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2218550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2218887Z res = mod(**inputs) 2025-12-04T09:39:42.2219250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2219647Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2220039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2220421Z outputs = block( 2025-12-04T09:39:42.2220756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2221124Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2221515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2221894Z return func(*args, **kwargs) 2025-12-04T09:39:42.2222272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2222684Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2223096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2223501Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2223918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2224340Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2224517Z 2025-12-04T09:39:42.2224620Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2224975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2225287Z res = mod(**inputs) 2025-12-04T09:39:42.2225640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2226036Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2226425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2226820Z outputs = block( 2025-12-04T09:39:42.2227145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2227510Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2227885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2228263Z return func(*args, **kwargs) 2025-12-04T09:39:42.2228640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2229067Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2229461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2229844Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2230192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2230657Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2230891Z 2025-12-04T09:39:42.2230997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2231351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2231663Z res = mod(**inputs) 2025-12-04T09:39:42.2232000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2232376Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2232741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2233106Z outputs = block( 2025-12-04T09:39:42.2233412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2233776Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2234148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2234508Z return func(*args, **kwargs) 2025-12-04T09:39:42.2234868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2235274Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2235671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2236055Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2236404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2236796Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2236963Z 2025-12-04T09:39:42.2237070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2237457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2237798Z res = mod(**inputs) 2025-12-04T09:39:42.2238146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2238528Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2238906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2239270Z outputs = block( 2025-12-04T09:39:42.2239583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2239929Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2240296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2240685Z return func(*args, **kwargs) 2025-12-04T09:39:42.2241049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:39:42.2241455Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:39:42.2241618Z 2025-12-04T09:39:42.2241717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2242068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2242371Z res = mod(**inputs) 2025-12-04T09:39:42.2242717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2243102Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2243480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2243840Z outputs = block( 2025-12-04T09:39:42.2244157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2244526Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2244899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2245279Z return func(*args, **kwargs) 2025-12-04T09:39:42.2245653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2246052Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2246430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2246801Z return func(*args, **kwargs) 2025-12-04T09:39:42.2247170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2247662Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2248139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2248543Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2248712Z 2025-12-04T09:39:42.2248801Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2249036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2249393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2249719Z res = mod(**inputs) 2025-12-04T09:39:42.2250077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2250467Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2250864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2251409Z outputs = block( 2025-12-04T09:39:42.2251795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2252206Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2252627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2253021Z return func(*args, **kwargs) 2025-12-04T09:39:42.2253411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2253809Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2254197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2254592Z return func(*args, **kwargs) 2025-12-04T09:39:42.2254965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2255375Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2255820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2256291Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2256480Z 2025-12-04T09:39:42.2256580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2256939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2257257Z res = mod(**inputs) 2025-12-04T09:39:42.2257608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2258002Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2258392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2258753Z outputs = block( 2025-12-04T09:39:42.2259074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2259435Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2259818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2260181Z return func(*args, **kwargs) 2025-12-04T09:39:42.2260551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2260942Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2261320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2261690Z return func(*args, **kwargs) 2025-12-04T09:39:42.2262061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2262453Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2262801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2263202Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2263378Z 2025-12-04T09:39:42.2263481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2263839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2264150Z res = mod(**inputs) 2025-12-04T09:39:42.2264502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2264899Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2265329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2265708Z outputs = block( 2025-12-04T09:39:42.2266019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2266375Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2266733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2267095Z return func(*args, **kwargs) 2025-12-04T09:39:42.2267450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2267846Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2268260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2268641Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2268989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2269377Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2269549Z 2025-12-04T09:39:42.2269650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2269994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2270302Z res = mod(**inputs) 2025-12-04T09:39:42.2270637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2271019Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2271395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2271748Z outputs = block( 2025-12-04T09:39:42.2272063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2272533Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2272906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2273261Z return func(*args, **kwargs) 2025-12-04T09:39:42.2273633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2274032Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2274435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2274804Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2275151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2275603Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2275874Z 2025-12-04T09:39:42.2275985Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2276344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2276657Z res = mod(**inputs) 2025-12-04T09:39:42.2277006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2277383Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2277765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2278128Z outputs = block( 2025-12-04T09:39:42.2278446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2278878Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2279278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2279641Z return func(*args, **kwargs) 2025-12-04T09:39:42.2279999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2280395Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2280789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2281174Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2281531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2281938Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2282096Z 2025-12-04T09:39:42.2282205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2282538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2282844Z res = mod(**inputs) 2025-12-04T09:39:42.2283180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2283553Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2283912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2284264Z outputs = block( 2025-12-04T09:39:42.2284571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2284920Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2285282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2285653Z return func(*args, **kwargs) 2025-12-04T09:39:42.2286006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2286374Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2286740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2287093Z return func(*args, **kwargs) 2025-12-04T09:39:42.2287451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2287917Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2288367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2288752Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2288921Z 2025-12-04T09:39:42.2289007Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2289235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2289580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2289891Z res = mod(**inputs) 2025-12-04T09:39:42.2290228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2290615Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2291006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2291432Z outputs = block( 2025-12-04T09:39:42.2291771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2292222Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2292625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2293031Z return func(*args, **kwargs) 2025-12-04T09:39:42.2293425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2293847Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2294254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2294634Z return func(*args, **kwargs) 2025-12-04T09:39:42.2295013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2295431Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2295860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2296333Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2296515Z 2025-12-04T09:39:42.2296614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2296964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2297268Z res = mod(**inputs) 2025-12-04T09:39:42.2297617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2298008Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2298395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2298759Z outputs = block( 2025-12-04T09:39:42.2299074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2299427Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2299786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2300149Z return func(*args, **kwargs) 2025-12-04T09:39:42.2300506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2300897Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2301287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2301647Z return func(*args, **kwargs) 2025-12-04T09:39:42.2302007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2302379Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2302728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2303117Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2303282Z 2025-12-04T09:39:42.2303391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2303736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2304052Z res = mod(**inputs) 2025-12-04T09:39:42.2304400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2304785Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2305155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2305518Z outputs = block( 2025-12-04T09:39:42.2305868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2306230Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2306599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2306960Z return func(*args, **kwargs) 2025-12-04T09:39:42.2307315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2307703Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2308097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2308472Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2308809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2309210Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2309384Z 2025-12-04T09:39:42.2309486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2309830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2310135Z res = mod(**inputs) 2025-12-04T09:39:42.2310482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2310863Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2311234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2311585Z outputs = block( 2025-12-04T09:39:42.2311897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2312258Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2312639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2313004Z return func(*args, **kwargs) 2025-12-04T09:39:42.2313363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2313759Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2314144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2314523Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2314868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2315304Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2315529Z 2025-12-04T09:39:42.2315627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2315983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2316297Z res = mod(**inputs) 2025-12-04T09:39:42.2316640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2317032Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2317403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2317757Z outputs = block( 2025-12-04T09:39:42.2318060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2318413Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2318782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2319137Z return func(*args, **kwargs) 2025-12-04T09:39:42.2319528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2319945Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2320348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2320729Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2321090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2321485Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2321649Z 2025-12-04T09:39:42.2321761Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2322121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2322432Z res = mod(**inputs) 2025-12-04T09:39:42.2322778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2323155Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2323526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2323883Z outputs = block( 2025-12-04T09:39:42.2324192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2324535Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2324902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2325267Z return func(*args, **kwargs) 2025-12-04T09:39:42.2325630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:39:42.2326042Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:39:42.2326211Z 2025-12-04T09:39:42.2326313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2326668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2326983Z res = mod(**inputs) 2025-12-04T09:39:42.2327337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2327729Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2328113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2328488Z outputs = block( 2025-12-04T09:39:42.2328824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2329209Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2329602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2329997Z return func(*args, **kwargs) 2025-12-04T09:39:42.2330387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2330803Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2331209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2331708Z return func(*args, **kwargs) 2025-12-04T09:39:42.2332116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2332659Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2333193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2333603Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2333771Z 2025-12-04T09:39:42.2333861Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2334090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2334434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2334748Z res = mod(**inputs) 2025-12-04T09:39:42.2335100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2335486Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2335873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2336267Z outputs = block( 2025-12-04T09:39:42.2336581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2336946Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2337318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2337690Z return func(*args, **kwargs) 2025-12-04T09:39:42.2338049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2338446Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2338832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2339201Z return func(*args, **kwargs) 2025-12-04T09:39:42.2339562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2339970Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2340414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2340888Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2341075Z 2025-12-04T09:39:42.2341175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2341530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2341845Z res = mod(**inputs) 2025-12-04T09:39:42.2342192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2342587Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2342972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2343336Z outputs = block( 2025-12-04T09:39:42.2343658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2344021Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2344398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2344764Z return func(*args, **kwargs) 2025-12-04T09:39:42.2345129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2345519Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2345903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2346266Z return func(*args, **kwargs) 2025-12-04T09:39:42.2346633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2347087Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2347454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2347836Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2348005Z 2025-12-04T09:39:42.2348102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2348442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2348736Z res = mod(**inputs) 2025-12-04T09:39:42.2349075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2349454Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2349839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2350194Z outputs = block( 2025-12-04T09:39:42.2350501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2350848Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2351203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2351559Z return func(*args, **kwargs) 2025-12-04T09:39:42.2351908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2352297Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2352680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2353055Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2353399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2353770Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2353938Z 2025-12-04T09:39:42.2354036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2354378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2354682Z res = mod(**inputs) 2025-12-04T09:39:42.2355013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2355390Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2355760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2356115Z outputs = block( 2025-12-04T09:39:42.2356420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2356768Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2357131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2357479Z return func(*args, **kwargs) 2025-12-04T09:39:42.2357832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2358218Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2358601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2358959Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2359293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2359729Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2359990Z 2025-12-04T09:39:42.2360098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2360458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2360763Z res = mod(**inputs) 2025-12-04T09:39:42.2361098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2361463Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2361829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2362187Z outputs = block( 2025-12-04T09:39:42.2362496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2362865Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2363236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2363598Z return func(*args, **kwargs) 2025-12-04T09:39:42.2363947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2364343Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2364768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2365150Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2365493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2365881Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2366049Z 2025-12-04T09:39:42.2366155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2366505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2366814Z res = mod(**inputs) 2025-12-04T09:39:42.2367156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2367540Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2367906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2368262Z outputs = block( 2025-12-04T09:39:42.2368574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2368922Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2369280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2369644Z return func(*args, **kwargs) 2025-12-04T09:39:42.2370002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2370381Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2370758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2371118Z return func(*args, **kwargs) 2025-12-04T09:39:42.2371643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2372156Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2372790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2373220Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2373411Z 2025-12-04T09:39:42.2373501Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2373779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2374160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2374486Z res = mod(**inputs) 2025-12-04T09:39:42.2374854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2375276Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2375678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2376052Z outputs = block( 2025-12-04T09:39:42.2376355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2376756Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2377123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2377477Z return func(*args, **kwargs) 2025-12-04T09:39:42.2377838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2378219Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2378594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2378948Z return func(*args, **kwargs) 2025-12-04T09:39:42.2379305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2379714Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2380149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2380616Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2380798Z 2025-12-04T09:39:42.2380902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2381266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2381609Z res = mod(**inputs) 2025-12-04T09:39:42.2381946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2382321Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2382689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2383030Z outputs = block( 2025-12-04T09:39:42.2383334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2383682Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2384034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2384386Z return func(*args, **kwargs) 2025-12-04T09:39:42.2384731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2385105Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2385468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2385817Z return func(*args, **kwargs) 2025-12-04T09:39:42.2386163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2386530Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2386859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2387286Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2387473Z 2025-12-04T09:39:42.2387578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2387908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2388212Z res = mod(**inputs) 2025-12-04T09:39:42.2388551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2388921Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2389283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2389634Z outputs = block( 2025-12-04T09:39:42.2389939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2390306Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2390664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2391014Z return func(*args, **kwargs) 2025-12-04T09:39:42.2391357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2391735Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2392116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2392484Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2392818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2393185Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2393351Z 2025-12-04T09:39:42.2393447Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2393785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2394078Z res = mod(**inputs) 2025-12-04T09:39:42.2394421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2394799Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2395173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2395536Z outputs = block( 2025-12-04T09:39:42.2395840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2396179Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2396529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2396886Z return func(*args, **kwargs) 2025-12-04T09:39:42.2397236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2397622Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2397998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2398371Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2398709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2399152Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2399374Z 2025-12-04T09:39:42.2399474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2399825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2400232Z res = mod(**inputs) 2025-12-04T09:39:42.2400589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2400999Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2401386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2401763Z outputs = block( 2025-12-04T09:39:42.2402064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2402422Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2402792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2403172Z return func(*args, **kwargs) 2025-12-04T09:39:42.2403530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2403927Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2404322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2404697Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2405052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2405443Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2405614Z 2025-12-04T09:39:42.2405724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2406072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2406393Z res = mod(**inputs) 2025-12-04T09:39:42.2406747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2407144Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2407523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2407893Z outputs = block( 2025-12-04T09:39:42.2408210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2408561Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2408933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2409305Z return func(*args, **kwargs) 2025-12-04T09:39:42.2409691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T09:39:42.2410120Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T09:39:42.2410297Z 2025-12-04T09:39:42.2410407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2410785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2411114Z res = mod(**inputs) 2025-12-04T09:39:42.2411559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2411987Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2412402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2412772Z outputs = block( 2025-12-04T09:39:42.2413096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2413459Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2413838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2414245Z return func(*args, **kwargs) 2025-12-04T09:39:42.2414629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2415015Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2415389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2415753Z return func(*args, **kwargs) 2025-12-04T09:39:42.2416115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T09:39:42.2416598Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T09:39:42.2417047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2417460Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2417627Z 2025-12-04T09:39:42.2417715Z cudagraph partition due to non gpu ops 2025-12-04T09:39:42.2417937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2418295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2418590Z res = mod(**inputs) 2025-12-04T09:39:42.2418921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2419275Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2419631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2419971Z outputs = block( 2025-12-04T09:39:42.2420262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2420594Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2420941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2421284Z return func(*args, **kwargs) 2025-12-04T09:39:42.2421614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2421977Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2422343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2422691Z return func(*args, **kwargs) 2025-12-04T09:39:42.2423032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T09:39:42.2423429Z attn_output, attn_weights = attention_interface( 2025-12-04T09:39:42.2423843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:39:42.2424281Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:39:42.2424454Z 2025-12-04T09:39:42.2424548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2424881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2425177Z res = mod(**inputs) 2025-12-04T09:39:42.2425498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2425860Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2426214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2426555Z outputs = block( 2025-12-04T09:39:42.2426839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2427205Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2427575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2427921Z return func(*args, **kwargs) 2025-12-04T09:39:42.2428267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T09:39:42.2428632Z attn_output, self_attn_weights = self.attn( 2025-12-04T09:39:42.2428995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2429339Z return func(*args, **kwargs) 2025-12-04T09:39:42.2429685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T09:39:42.2430083Z attn_output = self.c_proj(attn_output) 2025-12-04T09:39:42.2430456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2430840Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2431007Z 2025-12-04T09:39:42.2431105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2431447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2431747Z res = mod(**inputs) 2025-12-04T09:39:42.2432093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2432471Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2432842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2433191Z outputs = block( 2025-12-04T09:39:42.2433501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2433852Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2434208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2434611Z return func(*args, **kwargs) 2025-12-04T09:39:42.2434968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2435370Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2435769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T09:39:42.2436138Z hidden_states = self.c_fc(hidden_states) 2025-12-04T09:39:42.2436487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2436870Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2437047Z 2025-12-04T09:39:42.2437147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2437498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2437812Z res = mod(**inputs) 2025-12-04T09:39:42.2438151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2438535Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2438914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2439277Z outputs = block( 2025-12-04T09:39:42.2439580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2439935Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2440345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2440729Z return func(*args, **kwargs) 2025-12-04T09:39:42.2441090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2441492Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2441893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T09:39:42.2442261Z hidden_states = self.act(hidden_states) 2025-12-04T09:39:42.2442603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:39:42.2443047Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:39:42.2443290Z 2025-12-04T09:39:42.2443393Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2443732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2444045Z res = mod(**inputs) 2025-12-04T09:39:42.2444388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T09:39:42.2444763Z transformer_outputs = self.transformer( 2025-12-04T09:39:42.2445191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T09:39:42.2445551Z outputs = block( 2025-12-04T09:39:42.2445861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:39:42.2446203Z return super().__call__(*args, **kwargs) 2025-12-04T09:39:42.2446570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:39:42.2446938Z return func(*args, **kwargs) 2025-12-04T09:39:42.2447307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T09:39:42.2447721Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T09:39:42.2448129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T09:39:42.2448526Z hidden_states = self.c_proj(hidden_states) 2025-12-04T09:39:42.2448880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T09:39:42.2449275Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T09:39:42.2449442Z 2025-12-04T09:39:42.2449552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2449906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2450222Z res = mod(**inputs) 2025-12-04T09:39:42.2450577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1379, in forward 2025-12-04T09:39:42.2450965Z logits = self.score(hidden_states) 2025-12-04T09:39:42.2451094Z 2025-12-04T09:39:42.2451202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2451665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2452032Z res = mod(**inputs) 2025-12-04T09:39:42.2452429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1422, in forward 2025-12-04T09:39:42.2452895Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-12-04T09:39:42.2453096Z 2025-12-04T09:39:42.2453209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:39:42.2453597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:39:42.2454804Z res = mod(**inputs) 2025-12-04T09:39:42.2455205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1422, in forward 2025-12-04T09:39:42.2455727Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-12-04T09:39:42.2455935Z 2025-12-04T09:39:54.8066363Z Compilation time (from dynamo_timed): 18.54943985 2025-12-04T09:39:54.8066672Z pass 2025-12-04T09:39:54.8067563Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:39:54.8074005Z TIMING: _recursive_pre_grad_passes:0.0137 _recursive_joint_graph_passes:0.6073 _recursive_post_grad_passes:0.06628 async_compile.wait:0.73338 code_gen:9.58423 inductor_compile:10.77985 backend_compile:13.91813 gc:0.00098 entire_frame_compile:18.54944 total_wall_time:18.54944 2025-12-04T09:39:54.8075402Z STATS: call_* op count: 1126 | FakeTensorMode.__torch_dispatch__:7443 | FakeTensor.__torch_dispatch__:4239 | ProxyTorchDispatchMode.__torch_dispatch__:1681 2025-12-04T09:39:54.8075988Z Dynamo produced 2 graphs covering 1126 ops with 0 graph breaks (0 unique) 2025-12-04T09:39:57.3333904Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:39:57.3335115Z import pynvml # type: ignore[import] 2025-12-04T09:40:00.8987460Z 2025-12-04T09:40:01.7372977Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:40:01.7378269Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:40:01.7384665Z cpu eval GoogleFnet 2025-12-04T09:40:02.2079186Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:02.3957618Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:02.5714265Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:07.8144552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8145030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8145412Z res = mod(**inputs) 2025-12-04T09:40:07.8145855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8146287Z outputs = self.fnet( 2025-12-04T09:40:07.8146689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8147113Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8147562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8148028Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8148438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8148843Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8149261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8149716Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8150159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8150579Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8150994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8151453Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8151628Z 2025-12-04T09:40:07.8152764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8153233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8153694Z res = mod(**inputs) 2025-12-04T09:40:07.8154190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8154703Z outputs = self.fnet( 2025-12-04T09:40:07.8155099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8155518Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8155936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8156419Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8156975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8157376Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8157791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8158227Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8158665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8159115Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8159544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8160000Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8160179Z 2025-12-04T09:40:07.8160305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8160702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8161063Z res = mod(**inputs) 2025-12-04T09:40:07.8161453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8161866Z outputs = self.fnet( 2025-12-04T09:40:07.8162262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8162669Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8163075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8163502Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8163889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8164273Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8164691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8165143Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8165568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8165983Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8166438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8166872Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8167034Z 2025-12-04T09:40:07.8167145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8167521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8167879Z res = mod(**inputs) 2025-12-04T09:40:07.8168299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8168697Z outputs = self.fnet( 2025-12-04T09:40:07.8169101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8169530Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8169927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8170369Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8170768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8171172Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8171749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8172231Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8173024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8173435Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8173835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8174271Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8174433Z 2025-12-04T09:40:07.8174550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8174922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8175289Z res = mod(**inputs) 2025-12-04T09:40:07.8175663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8176061Z outputs = self.fnet( 2025-12-04T09:40:07.8176426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8176833Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8177229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8177649Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8178047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8178426Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8178830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8179264Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8179694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8180104Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8180510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8180946Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8181118Z 2025-12-04T09:40:07.8181237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8181614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8181952Z res = mod(**inputs) 2025-12-04T09:40:07.8182331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8182735Z outputs = self.fnet( 2025-12-04T09:40:07.8183114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8183520Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8183998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8184469Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8184867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8185251Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8185666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8186101Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8186526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8186948Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8187420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8187893Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8188060Z 2025-12-04T09:40:07.8188178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8188562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8188913Z res = mod(**inputs) 2025-12-04T09:40:07.8189297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8189695Z outputs = self.fnet( 2025-12-04T09:40:07.8190080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8190493Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8190898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8191331Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8191728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8192119Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8192523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8192970Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8193403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8193831Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8194243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8194666Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8194825Z 2025-12-04T09:40:07.8194943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8195318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8195644Z res = mod(**inputs) 2025-12-04T09:40:07.8196012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8196425Z outputs = self.fnet( 2025-12-04T09:40:07.8196782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8197202Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8197597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8198020Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8198401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8198820Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8199250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8199686Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8200111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8200528Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8200936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8201358Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8201528Z 2025-12-04T09:40:07.8201637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8202041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8202383Z res = mod(**inputs) 2025-12-04T09:40:07.8202749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8203140Z outputs = self.fnet( 2025-12-04T09:40:07.8203514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8203907Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8204299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8204712Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8205099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8205474Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8205880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8206304Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8206720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8207131Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8207534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8207958Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8208116Z 2025-12-04T09:40:07.8208224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8208597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8208936Z res = mod(**inputs) 2025-12-04T09:40:07.8209308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8209697Z outputs = self.fnet( 2025-12-04T09:40:07.8210071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8210473Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8210858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8211368Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8211782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8212167Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8212576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8213029Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8213511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8213939Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8214337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8214762Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8214920Z 2025-12-04T09:40:07.8215036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8215401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8215737Z res = mod(**inputs) 2025-12-04T09:40:07.8216107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8216525Z outputs = self.fnet( 2025-12-04T09:40:07.8216893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8217301Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8217694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8218113Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8218497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8218875Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8219276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8219800Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8220228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8220644Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8221030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8221429Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8221588Z 2025-12-04T09:40:07.8221691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8222063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8222406Z res = mod(**inputs) 2025-12-04T09:40:07.8222791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8223185Z outputs = self.fnet( 2025-12-04T09:40:07.8223568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8223962Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8224371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8224784Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8225159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8225551Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8225968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8226390Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8226812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8227237Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8227695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8228143Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8228324Z 2025-12-04T09:40:07.8228434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8228809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8229157Z res = mod(**inputs) 2025-12-04T09:40:07.8229526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8229928Z outputs = self.fnet( 2025-12-04T09:40:07.8230314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 511, in forward 2025-12-04T09:40:07.8230733Z embedding_output = self.embeddings( 2025-12-04T09:40:07.8231156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 141, in forward 2025-12-04T09:40:07.8231574Z embeddings = self.projection(embeddings) 2025-12-04T09:40:07.8231723Z 2025-12-04T09:40:07.8231816Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8232060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8232434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8232771Z res = mod(**inputs) 2025-12-04T09:40:07.8233141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8233537Z outputs = self.fnet( 2025-12-04T09:40:07.8233912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8234335Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8234728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8235162Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8235549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8235930Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8236326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8236768Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8237193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8237614Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8238015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8238453Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8238616Z 2025-12-04T09:40:07.8238737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8239105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8239443Z res = mod(**inputs) 2025-12-04T09:40:07.8239817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8240226Z outputs = self.fnet( 2025-12-04T09:40:07.8240591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8241008Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8241410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8241825Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8242207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8242630Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8243052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8243468Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8243893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8244300Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8244705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8245126Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8245295Z 2025-12-04T09:40:07.8245428Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8245809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8246152Z res = mod(**inputs) 2025-12-04T09:40:07.8246521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8246918Z outputs = self.fnet( 2025-12-04T09:40:07.8247294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8247691Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8248089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8248506Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8248898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8249279Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8249690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8250120Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8250539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8250952Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8251457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8251911Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8252078Z 2025-12-04T09:40:07.8252190Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8252586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8252930Z res = mod(**inputs) 2025-12-04T09:40:07.8253307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8253710Z outputs = self.fnet( 2025-12-04T09:40:07.8254097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8254511Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8254912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8255345Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8255745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8256138Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8256545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8256987Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8257467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8257951Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8258369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8258814Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8258981Z 2025-12-04T09:40:07.8259099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8259477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8259831Z res = mod(**inputs) 2025-12-04T09:40:07.8260212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8260642Z outputs = self.fnet( 2025-12-04T09:40:07.8261030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8261464Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8261868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8262289Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8262689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8263079Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8263493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8263914Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8264366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8264808Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8265236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8265716Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8266159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8266579Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8266724Z 2025-12-04T09:40:07.8266833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8267213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8267552Z res = mod(**inputs) 2025-12-04T09:40:07.8267924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8268311Z outputs = self.fnet( 2025-12-04T09:40:07.8268685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8269087Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8269475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8269886Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8270269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8270645Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8271041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8271455Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8271922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8272557Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8272986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8273472Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8273919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8274371Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8274780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8275289Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8275604Z 2025-12-04T09:40:07.8275725Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8276107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8276453Z res = mod(**inputs) 2025-12-04T09:40:07.8276831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8277229Z outputs = self.fnet( 2025-12-04T09:40:07.8277603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8278015Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8278416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8278839Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8279233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8279619Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8280032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8280419Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8280821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8281219Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8281617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8282119Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8282577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8282990Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8283135Z 2025-12-04T09:40:07.8283224Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8283480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8283855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8284198Z res = mod(**inputs) 2025-12-04T09:40:07.8284562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8284961Z outputs = self.fnet( 2025-12-04T09:40:07.8285338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8285734Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8286136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8286559Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8287005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8287408Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8287813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8288242Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8288663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8289072Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8289481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8289914Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8290098Z 2025-12-04T09:40:07.8290209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8290592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8290935Z res = mod(**inputs) 2025-12-04T09:40:07.8291420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8291835Z outputs = self.fnet( 2025-12-04T09:40:07.8292222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8292643Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8293042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8293440Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8293805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8294173Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8294551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8294958Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8295359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8295746Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8296127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8296532Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8296685Z 2025-12-04T09:40:07.8296795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8297146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8297471Z res = mod(**inputs) 2025-12-04T09:40:07.8297823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8298195Z outputs = self.fnet( 2025-12-04T09:40:07.8298539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8298918Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8299292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8299679Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8300045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8300408Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8300791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8301224Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8301643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8302031Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8302418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8302821Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8302980Z 2025-12-04T09:40:07.8303083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8303450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8303753Z res = mod(**inputs) 2025-12-04T09:40:07.8304119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8304486Z outputs = self.fnet( 2025-12-04T09:40:07.8304835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8305201Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8305564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8305946Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8306291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8306641Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8307012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8307406Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8307788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8308164Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8308537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8308933Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8309079Z 2025-12-04T09:40:07.8309178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8309523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8309836Z res = mod(**inputs) 2025-12-04T09:40:07.8310172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8310536Z outputs = self.fnet( 2025-12-04T09:40:07.8310881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8311256Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8311614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8311996Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8312351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8312703Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8313068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8313451Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8313842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8314221Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8314656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8315114Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8315526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8315904Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8316043Z 2025-12-04T09:40:07.8316144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8316490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8316806Z res = mod(**inputs) 2025-12-04T09:40:07.8317144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8317539Z outputs = self.fnet( 2025-12-04T09:40:07.8317878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8318234Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8318591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8318966Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8319309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8319643Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8320005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8320378Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8320748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8321123Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8321512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8321935Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8322322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8322715Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8323077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8323507Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8323724Z 2025-12-04T09:40:07.8323827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8324165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8324472Z res = mod(**inputs) 2025-12-04T09:40:07.8324800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8325150Z outputs = self.fnet( 2025-12-04T09:40:07.8325480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8325833Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8326178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8326552Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8326896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8327239Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8327628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8328015Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8328400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8328765Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8329162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8329615Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8330038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8330413Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8330571Z 2025-12-04T09:40:07.8330650Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8330889Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8331232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8331659Z res = mod(**inputs) 2025-12-04T09:40:07.8332048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8332463Z outputs = self.fnet( 2025-12-04T09:40:07.8332849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8333245Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8333615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8334003Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8334362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8334737Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8335155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8335602Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8336045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8336479Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8336899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8337350Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8337526Z 2025-12-04T09:40:07.8337639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8338032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8338385Z res = mod(**inputs) 2025-12-04T09:40:07.8338762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8339178Z outputs = self.fnet( 2025-12-04T09:40:07.8339566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8339992Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8340351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8340727Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8341074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8341412Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8341810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8342218Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8342590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8342962Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8343333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8343724Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8343876Z 2025-12-04T09:40:07.8343976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8344333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8344659Z res = mod(**inputs) 2025-12-04T09:40:07.8344995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8345352Z outputs = self.fnet( 2025-12-04T09:40:07.8345702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8346110Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8346457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8346833Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8347184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8347536Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8347910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8348296Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8348679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8349040Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8349407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8349792Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8349937Z 2025-12-04T09:40:07.8350268Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8350594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8350898Z res = mod(**inputs) 2025-12-04T09:40:07.8351233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8351593Z outputs = self.fnet( 2025-12-04T09:40:07.8351926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8352293Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8352647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8353012Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8353357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8353702Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8354064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8354441Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8354822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8355226Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8355594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8355991Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8356145Z 2025-12-04T09:40:07.8356243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8356582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8356879Z res = mod(**inputs) 2025-12-04T09:40:07.8357209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8357561Z outputs = self.fnet( 2025-12-04T09:40:07.8357891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8375407Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8376035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8376460Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8376829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8377202Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8377594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8377981Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8378398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8378794Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8379206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8379651Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8380069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8380460Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8380598Z 2025-12-04T09:40:07.8380716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8381076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8381399Z res = mod(**inputs) 2025-12-04T09:40:07.8381755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8382122Z outputs = self.fnet( 2025-12-04T09:40:07.8382482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8382865Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8383239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8383622Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8383985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8384343Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8384712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8385094Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8385486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8385883Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8386462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8386972Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8387390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8387806Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8388181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8388641Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8388871Z 2025-12-04T09:40:07.8388989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8389353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8389712Z res = mod(**inputs) 2025-12-04T09:40:07.8390069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8390441Z outputs = self.fnet( 2025-12-04T09:40:07.8390783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8391165Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8391530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8391918Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8392270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8392635Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8393041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8393433Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8393855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8394254Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8394661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8395118Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8395556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8395936Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8396069Z 2025-12-04T09:40:07.8396157Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8396392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8396747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8397068Z res = mod(**inputs) 2025-12-04T09:40:07.8397406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8397774Z outputs = self.fnet( 2025-12-04T09:40:07.8398121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8398499Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8398864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8399251Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8399614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8399975Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8400413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8400829Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8401219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8401590Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8401961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8402357Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8402512Z 2025-12-04T09:40:07.8402627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8402976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8403320Z res = mod(**inputs) 2025-12-04T09:40:07.8403674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8404044Z outputs = self.fnet( 2025-12-04T09:40:07.8404398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8404783Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8405174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8405586Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8405976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8406368Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8406767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8407200Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8407626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8408032Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8408441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8408876Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8409038Z 2025-12-04T09:40:07.8409158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8409529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8409871Z res = mod(**inputs) 2025-12-04T09:40:07.8410243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8410643Z outputs = self.fnet( 2025-12-04T09:40:07.8411014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8411516Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8411918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8412352Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8412770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8413164Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8413623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8414025Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8414470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8414914Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8415341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8415780Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8415952Z 2025-12-04T09:40:07.8416063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8416447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8416790Z res = mod(**inputs) 2025-12-04T09:40:07.8417174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8417575Z outputs = self.fnet( 2025-12-04T09:40:07.8417969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8418372Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8418774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8419279Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8419638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8420001Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8420381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8420783Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8421176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8421568Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8421954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8422361Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8422513Z 2025-12-04T09:40:07.8422614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8422969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8423291Z res = mod(**inputs) 2025-12-04T09:40:07.8423633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8424012Z outputs = self.fnet( 2025-12-04T09:40:07.8424366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8424749Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8425119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8425515Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8425883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8426236Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8426618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8427012Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8427418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8427810Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8428227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8428687Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8429147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8429551Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8429695Z 2025-12-04T09:40:07.8429800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8430159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8430470Z res = mod(**inputs) 2025-12-04T09:40:07.8430825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8431197Z outputs = self.fnet( 2025-12-04T09:40:07.8431547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8431935Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8432310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8432704Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8433059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8433419Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8433800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8434187Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8434578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8434972Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8435379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8435829Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8436238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8436654Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8437034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8437487Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8437723Z 2025-12-04T09:40:07.8437826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8438185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8438507Z res = mod(**inputs) 2025-12-04T09:40:07.8438859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8439223Z outputs = self.fnet( 2025-12-04T09:40:07.8439566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8439932Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8440282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8440662Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8441013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8441353Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8441722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8442102Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8442532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8442928Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8443321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8443776Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8444224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8444601Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8444740Z 2025-12-04T09:40:07.8444820Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8445052Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8445415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8445728Z res = mod(**inputs) 2025-12-04T09:40:07.8446074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8446440Z outputs = self.fnet( 2025-12-04T09:40:07.8446778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8447145Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8447515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8447891Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8448246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8448601Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8448973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8449363Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8449755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8450133Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8450521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8450922Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8451082Z 2025-12-04T09:40:07.8451185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8451693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8452045Z res = mod(**inputs) 2025-12-04T09:40:07.8452438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8452870Z outputs = self.fnet( 2025-12-04T09:40:07.8453241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8453612Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8453993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8454385Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8454742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8455107Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8455498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8455911Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8456350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8456758Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8457144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8457549Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8457700Z 2025-12-04T09:40:07.8457813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8458159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8458473Z res = mod(**inputs) 2025-12-04T09:40:07.8458815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8459208Z outputs = self.fnet( 2025-12-04T09:40:07.8459561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8459939Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8460301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8460690Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8461055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8461411Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8461794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8462190Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8462587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8462975Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8463359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8463760Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8463917Z 2025-12-04T09:40:07.8464021Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8464378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8464689Z res = mod(**inputs) 2025-12-04T09:40:07.8465035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8465407Z outputs = self.fnet( 2025-12-04T09:40:07.8465753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8466127Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8466500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8466891Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8467244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8467602Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8467977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8468375Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8468765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8469148Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8469530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8469974Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8470146Z 2025-12-04T09:40:07.8470250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8470612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8470933Z res = mod(**inputs) 2025-12-04T09:40:07.8471276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8471655Z outputs = self.fnet( 2025-12-04T09:40:07.8472007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8472527Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8472917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8473392Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8473790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8474162Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8474545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8474941Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8475370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8475789Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8476226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8476708Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8477157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8477571Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8477723Z 2025-12-04T09:40:07.8477834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8478214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8478547Z res = mod(**inputs) 2025-12-04T09:40:07.8478914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8479309Z outputs = self.fnet( 2025-12-04T09:40:07.8479682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8480076Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8480472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8480895Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8481282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8481656Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8482065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8482479Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8482893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8483311Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8483743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8484202Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8484654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8485101Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8485481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8485942Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8486176Z 2025-12-04T09:40:07.8486280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8486637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8486956Z res = mod(**inputs) 2025-12-04T09:40:07.8487302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8487702Z outputs = self.fnet( 2025-12-04T09:40:07.8488062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8488447Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8488816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8489220Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8489582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8489935Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8490319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8490709Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8491127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8491627Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8492094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8492599Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8493056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8493462Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8493619Z 2025-12-04T09:40:07.8493702Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8493943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8494295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8494622Z res = mod(**inputs) 2025-12-04T09:40:07.8494998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8495398Z outputs = self.fnet( 2025-12-04T09:40:07.8495770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8496172Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8496564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8496972Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8497331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8497687Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8498068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8498468Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8498917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8499326Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8499710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8500110Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8500272Z 2025-12-04T09:40:07.8500376Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8500730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8501041Z res = mod(**inputs) 2025-12-04T09:40:07.8501391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8501786Z outputs = self.fnet( 2025-12-04T09:40:07.8502139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8502528Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8502890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8503282Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8503644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8503996Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8504380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8504780Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8505184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8505563Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8505954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8506360Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8506513Z 2025-12-04T09:40:07.8506622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8506970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8507288Z res = mod(**inputs) 2025-12-04T09:40:07.8507637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8508003Z outputs = self.fnet( 2025-12-04T09:40:07.8508356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8508737Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8509111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8509499Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8509862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8510218Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8510590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8510993Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8511390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8511775Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8512185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8512615Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8512775Z 2025-12-04T09:40:07.8512879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8513237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8513601Z res = mod(**inputs) 2025-12-04T09:40:07.8513945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8514309Z outputs = self.fnet( 2025-12-04T09:40:07.8514644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8515016Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8515401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8515784Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8516131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8516478Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8516847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8517240Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8517620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8517995Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8518369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8518762Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8518914Z 2025-12-04T09:40:07.8519016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8519360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8519669Z res = mod(**inputs) 2025-12-04T09:40:07.8520001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8520359Z outputs = self.fnet( 2025-12-04T09:40:07.8520697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8521058Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8521418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8521801Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8522158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8522504Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8522881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8523276Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8523686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8524060Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8524455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8524893Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8525291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8525707Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8525864Z 2025-12-04T09:40:07.8525964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8526307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8526611Z res = mod(**inputs) 2025-12-04T09:40:07.8526952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8527321Z outputs = self.fnet( 2025-12-04T09:40:07.8527662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8528039Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8528409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8528819Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8529176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8529539Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8529917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8530301Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8530690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8531083Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8531606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8532100Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8532575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8533034Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8533440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8533916Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8534167Z 2025-12-04T09:40:07.8534279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8534655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8534980Z res = mod(**inputs) 2025-12-04T09:40:07.8535326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8535706Z outputs = self.fnet( 2025-12-04T09:40:07.8536062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8536440Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8536816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8537212Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8537573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8537927Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8538314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8538707Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8539108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8539496Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8539941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8540423Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8540848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8541239Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8541379Z 2025-12-04T09:40:07.8541459Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8541696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8542043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8542367Z res = mod(**inputs) 2025-12-04T09:40:07.8542748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8543120Z outputs = self.fnet( 2025-12-04T09:40:07.8543474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8543856Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8544228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8544615Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8544976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8545337Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8545705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8546091Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8546484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8546860Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8547223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8547621Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8547781Z 2025-12-04T09:40:07.8547882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8548228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8548532Z res = mod(**inputs) 2025-12-04T09:40:07.8548873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8549235Z outputs = self.fnet( 2025-12-04T09:40:07.8549568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8549944Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8550309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8550697Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8551043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8551395Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8551766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8552156Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8552538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8552913Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8553317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8553730Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8553890Z 2025-12-04T09:40:07.8553991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8554334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8554645Z res = mod(**inputs) 2025-12-04T09:40:07.8554984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8555349Z outputs = self.fnet( 2025-12-04T09:40:07.8555694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8556085Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8556445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8556830Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8557046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8557128Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8557370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8557463Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8557712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8557791Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8558037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8558135Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8558140Z 2025-12-04T09:40:07.8558241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8558440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8558503Z res = mod(**inputs) 2025-12-04T09:40:07.8558749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8558811Z outputs = self.fnet( 2025-12-04T09:40:07.8559050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8559127Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8559368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8559451Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8559676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8559752Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8559999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8560092Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8560333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8560416Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8560654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8560755Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8560759Z 2025-12-04T09:40:07.8560857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8561089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8561173Z res = mod(**inputs) 2025-12-04T09:40:07.8561415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8561476Z outputs = self.fnet( 2025-12-04T09:40:07.8561715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8561785Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8562026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8562106Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8562345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8562428Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8562662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8562749Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8562998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8563069Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8563340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8563449Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8563681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8563767Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8563770Z 2025-12-04T09:40:07.8563869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8564061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8564121Z res = mod(**inputs) 2025-12-04T09:40:07.8564353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8564419Z outputs = self.fnet( 2025-12-04T09:40:07.8564652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8564727Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8564960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8565041Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8565260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8565335Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8565568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8565654Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8565906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8565988Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8566258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8566366Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8566614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8566754Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8566965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8567168Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8567171Z 2025-12-04T09:40:07.8567273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8567470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8567530Z res = mod(**inputs) 2025-12-04T09:40:07.8567771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8567841Z outputs = self.fnet( 2025-12-04T09:40:07.8568080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8568177Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8568423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8568506Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8568736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8568811Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8569053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8569146Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8569409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8569490Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8569768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8569894Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8570144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8570223Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8570226Z 2025-12-04T09:40:07.8570311Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8570411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8570603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8570674Z res = mod(**inputs) 2025-12-04T09:40:07.8570916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8570981Z outputs = self.fnet( 2025-12-04T09:40:07.8571305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8571404Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8571685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8571778Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8572020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8572112Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8572590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8572705Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8572973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8573128Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8573434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8573541Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8573546Z 2025-12-04T09:40:07.8573653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8573876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8573936Z res = mod(**inputs) 2025-12-04T09:40:07.8574183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8574247Z outputs = self.fnet( 2025-12-04T09:40:07.8574485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8574590Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8574837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8574929Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8575146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8575221Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8575470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8575561Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8575801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8575887Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8576132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8576236Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8576240Z 2025-12-04T09:40:07.8576337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8576528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8576594Z res = mod(**inputs) 2025-12-04T09:40:07.8576881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8576948Z outputs = self.fnet( 2025-12-04T09:40:07.8577192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8577260Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8577509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8577590Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8577807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8577888Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8578128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8578227Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8578469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8578545Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8578794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8578890Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8578894Z 2025-12-04T09:40:07.8579031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8579237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8579297Z res = mod(**inputs) 2025-12-04T09:40:07.8579542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8579604Z outputs = self.fnet( 2025-12-04T09:40:07.8579841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8579916Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8580155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8580269Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8580485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8580562Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8580804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8580896Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8581136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8581220Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8581459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8581563Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8581567Z 2025-12-04T09:40:07.8581668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8581862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8581933Z res = mod(**inputs) 2025-12-04T09:40:07.8582174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8582243Z outputs = self.fnet( 2025-12-04T09:40:07.8582481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8582550Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8582796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8582878Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8583090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8583174Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8583414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8583502Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8583752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8583825Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8584097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8584208Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8584450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8584528Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8584533Z 2025-12-04T09:40:07.8584631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8584885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8584966Z res = mod(**inputs) 2025-12-04T09:40:07.8585204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8585275Z outputs = self.fnet( 2025-12-04T09:40:07.8585510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8585584Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8585820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8585899Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8586119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8586213Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8586453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8586540Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8586792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8586871Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8587136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8587244Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8587488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8587595Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8587813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8587989Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8587993Z 2025-12-04T09:40:07.8588093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8588291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8588352Z res = mod(**inputs) 2025-12-04T09:40:07.8588597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8588658Z outputs = self.fnet( 2025-12-04T09:40:07.8588894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8588971Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8589210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8589293Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8589514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8589589Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8589832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8589910Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8590156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8590237Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8590505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8591308Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8591574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8591652Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8591656Z 2025-12-04T09:40:07.8591742Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8591841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8592033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8592101Z res = mod(**inputs) 2025-12-04T09:40:07.8592339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8592417Z outputs = self.fnet( 2025-12-04T09:40:07.8592671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8592741Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8592984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8593065Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8593280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8593362Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8593599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8593709Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8593938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8594014Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8594258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8594358Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8594361Z 2025-12-04T09:40:07.8594466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8594653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8594713Z res = mod(**inputs) 2025-12-04T09:40:07.8594956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8595016Z outputs = self.fnet( 2025-12-04T09:40:07.8595250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8595329Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8595564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8595650Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8595860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8595933Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8596175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8596265Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8596507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8596584Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8596815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8596965Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8596969Z 2025-12-04T09:40:07.8597082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8597272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8597342Z res = mod(**inputs) 2025-12-04T09:40:07.8597580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8597648Z outputs = self.fnet( 2025-12-04T09:40:07.8597883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8597951Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8598196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8598292Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8598508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8598590Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8598820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8598915Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8599144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8599219Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8599460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8599554Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8599559Z 2025-12-04T09:40:07.8599663Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8599856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8599916Z res = mod(**inputs) 2025-12-04T09:40:07.8600154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8600214Z outputs = self.fnet( 2025-12-04T09:40:07.8600444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8600519Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8600749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8600834Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8601040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8601114Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8601353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8601445Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8601683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8601758Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8601989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8602089Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8602092Z 2025-12-04T09:40:07.8602189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8602376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8602444Z res = mod(**inputs) 2025-12-04T09:40:07.8602707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8602795Z outputs = self.fnet( 2025-12-04T09:40:07.8603026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8603094Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8603334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8603415Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8603631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8603704Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8603957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8604045Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8604296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8604368Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8604646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8604757Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8605004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8605081Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8605085Z 2025-12-04T09:40:07.8605184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8605382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8605445Z res = mod(**inputs) 2025-12-04T09:40:07.8605690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8605753Z outputs = self.fnet( 2025-12-04T09:40:07.8605990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8606068Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8606304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8606384Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8606606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8606682Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8606928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8607010Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8607261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8607340Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8607610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8607721Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8607966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8608070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8608286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8608496Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8608515Z 2025-12-04T09:40:07.8608617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8608818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8608878Z res = mod(**inputs) 2025-12-04T09:40:07.8609124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8609188Z outputs = self.fnet( 2025-12-04T09:40:07.8609425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8609503Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8609760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8609853Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8610071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8610146Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8610393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8610474Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8610730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8610812Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8611083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8611217Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8611546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8611634Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8611639Z 2025-12-04T09:40:07.8611729Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8611833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8612039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8612104Z res = mod(**inputs) 2025-12-04T09:40:07.8612367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8612445Z outputs = self.fnet( 2025-12-04T09:40:07.8612708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8612785Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8613057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8613148Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8613391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8613475Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8613730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8613833Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8614073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8614150Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8614399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8614536Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8614556Z 2025-12-04T09:40:07.8614665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8614857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8614920Z res = mod(**inputs) 2025-12-04T09:40:07.8615178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8615241Z outputs = self.fnet( 2025-12-04T09:40:07.8615489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8615559Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8615802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8615913Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8616126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8616204Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8616449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8616543Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8616789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8616867Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8617106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8617211Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8617214Z 2025-12-04T09:40:07.8617316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8617515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8617575Z res = mod(**inputs) 2025-12-04T09:40:07.8617816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8617886Z outputs = self.fnet( 2025-12-04T09:40:07.8618134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8618204Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8618451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8618531Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8618754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8618829Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8619068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8619167Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8619403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8619478Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8619723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8619817Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8619821Z 2025-12-04T09:40:07.8619925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8620118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8620219Z res = mod(**inputs) 2025-12-04T09:40:07.8620483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8620544Z outputs = self.fnet( 2025-12-04T09:40:07.8620790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8620859Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8621098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8621189Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8621404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8621494Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8621746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8621839Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8622087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8622165Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8622406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8622508Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8622511Z 2025-12-04T09:40:07.8622610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8622805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8622868Z res = mod(**inputs) 2025-12-04T09:40:07.8623112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8623184Z outputs = self.fnet( 2025-12-04T09:40:07.8623427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8623495Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8623744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8623823Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8624044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8624118Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8624361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8624448Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8624705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8624787Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8625062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8625173Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8625429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8625507Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8625510Z 2025-12-04T09:40:07.8625607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8625800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8625861Z res = mod(**inputs) 2025-12-04T09:40:07.8626142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8626220Z outputs = self.fnet( 2025-12-04T09:40:07.8626454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8626529Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8626760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8626840Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8627053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8627126Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8627384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8627464Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8627707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8627785Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8628044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8628158Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8628391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8628500Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8628710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8628881Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8628886Z 2025-12-04T09:40:07.8628989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8629175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8629234Z res = mod(**inputs) 2025-12-04T09:40:07.8629473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8629533Z outputs = self.fnet( 2025-12-04T09:40:07.8629767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8629844Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8630077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8630165Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8630374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8630449Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8630686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8630763Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8631004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8631082Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8631342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8631466Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8631699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8631805Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8631824Z 2025-12-04T09:40:07.8631910Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8632005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8632198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8632256Z res = mod(**inputs) 2025-12-04T09:40:07.8632492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8632561Z outputs = self.fnet( 2025-12-04T09:40:07.8632796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8632883Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8633130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8633210Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8633432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8633505Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8633748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8633847Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8634091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8634174Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8634420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8634519Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8634526Z 2025-12-04T09:40:07.8634632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8634830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8634890Z res = mod(**inputs) 2025-12-04T09:40:07.8635201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8635261Z outputs = self.fnet( 2025-12-04T09:40:07.8635504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8635571Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8635807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8635894Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8636109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8636190Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8636428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8636517Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8636763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8636839Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8637077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8637178Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8637184Z 2025-12-04T09:40:07.8637280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8637507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8637584Z res = mod(**inputs) 2025-12-04T09:40:07.8637815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8637884Z outputs = self.fnet( 2025-12-04T09:40:07.8638124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8638193Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8638442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8638522Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8638742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8638847Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8639094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8639197Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8639445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8639530Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8639779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8639878Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8639882Z 2025-12-04T09:40:07.8639991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8640189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8640255Z res = mod(**inputs) 2025-12-04T09:40:07.8640515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8640582Z outputs = self.fnet( 2025-12-04T09:40:07.8640837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8640909Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8641157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8641247Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8641467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8641552Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8641802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8641900Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8642159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8642238Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8642485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8642592Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8642596Z 2025-12-04T09:40:07.8642697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8642900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8642963Z res = mod(**inputs) 2025-12-04T09:40:07.8643212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8643286Z outputs = self.fnet( 2025-12-04T09:40:07.8643573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8643669Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8643916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8644000Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8644224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8644299Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8644541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8644651Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8644915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8644996Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8645272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8645384Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8645638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8645719Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8645722Z 2025-12-04T09:40:07.8645829Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8646024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8646088Z res = mod(**inputs) 2025-12-04T09:40:07.8646339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8646405Z outputs = self.fnet( 2025-12-04T09:40:07.8646653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8646732Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8646977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8647067Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8647289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8647366Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8647617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8647701Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8647963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8648046Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8648318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8648436Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8648683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8648790Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8649014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8649190Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8649195Z 2025-12-04T09:40:07.8649334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8649546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8649608Z res = mod(**inputs) 2025-12-04T09:40:07.8649859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8649923Z outputs = self.fnet( 2025-12-04T09:40:07.8650171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8650240Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8650481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8650568Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8650801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8650878Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8651137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8651222Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8651572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8651658Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8651949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8652092Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8652350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8652447Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8652454Z 2025-12-04T09:40:07.8652541Z cudagraph partition due to non gpu ops 2025-12-04T09:40:07.8652653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8652878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8652939Z res = mod(**inputs) 2025-12-04T09:40:07.8653186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8653259Z outputs = self.fnet( 2025-12-04T09:40:07.8653502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8653580Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8653825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8653911Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8654144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8654225Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8654469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8654572Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8654814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8654901Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8655145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8655249Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8655252Z 2025-12-04T09:40:07.8655371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8655598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8655684Z res = mod(**inputs) 2025-12-04T09:40:07.8655932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8655996Z outputs = self.fnet( 2025-12-04T09:40:07.8656244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8656314Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8656557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8656647Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8656877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8656963Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8657201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8657295Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8657538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8657611Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8657857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8657954Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8657957Z 2025-12-04T09:40:07.8658054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8658256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8658318Z res = mod(**inputs) 2025-12-04T09:40:07.8658555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8658629Z outputs = self.fnet( 2025-12-04T09:40:07.8658865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8658941Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8659179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8659258Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8659480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8659557Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8659795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8659895Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8660145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8660229Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8660467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8660564Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8660568Z 2025-12-04T09:40:07.8660674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8660864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8660933Z res = mod(**inputs) 2025-12-04T09:40:07.8661172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8661265Z outputs = self.fnet( 2025-12-04T09:40:07.8661536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8661605Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8661841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8661929Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8662143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8662225Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8662458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T09:40:07.8662571Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T09:40:07.8662815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T09:40:07.8662892Z self_outputs = self.self(hidden_states) 2025-12-04T09:40:07.8663135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T09:40:07.8663229Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T09:40:07.8663232Z 2025-12-04T09:40:07.8663328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8663520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8663580Z res = mod(**inputs) 2025-12-04T09:40:07.8663815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8663884Z outputs = self.fnet( 2025-12-04T09:40:07.8664122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8664199Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8664434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8664513Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8664732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8664805Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8665048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8665125Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8665376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8665456Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8665724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8665833Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8666075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T09:40:07.8666153Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8666156Z 2025-12-04T09:40:07.8666258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8666446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8666506Z res = mod(**inputs) 2025-12-04T09:40:07.8666747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8666808Z outputs = self.fnet( 2025-12-04T09:40:07.8667082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8667170Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8667408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8667498Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8667709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8667783Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8668029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8668108Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8668383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8668459Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8668725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T09:40:07.8668842Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T09:40:07.8669081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T09:40:07.8669185Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:07.8669399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T09:40:07.8669577Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T09:40:07.8669582Z 2025-12-04T09:40:07.8669693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8669887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8669950Z res = mod(**inputs) 2025-12-04T09:40:07.8670195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T09:40:07.8670259Z outputs = self.fnet( 2025-12-04T09:40:07.8670503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T09:40:07.8670571Z encoder_outputs = self.encoder( 2025-12-04T09:40:07.8670808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T09:40:07.8670899Z layer_outputs = layer_module(hidden_states) 2025-12-04T09:40:07.8671111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:07.8671187Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:07.8671435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T09:40:07.8671518Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:07.8671774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:07.8671846Z return forward_fn(*input_tensors) 2025-12-04T09:40:07.8672110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T09:40:07.8672240Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T09:40:07.8672654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T09:40:07.8672745Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8672749Z 2025-12-04T09:40:07.8672851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8673112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8673210Z res = mod(**inputs) 2025-12-04T09:40:07.8673456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 680, in forward 2025-12-04T09:40:07.8673556Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:40:07.8673802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 358, in forward 2025-12-04T09:40:07.8673914Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:40:07.8674160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 339, in forward 2025-12-04T09:40:07.8674297Z hidden_states = self.transform(hidden_states) 2025-12-04T09:40:07.8674540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 320, in forward 2025-12-04T09:40:07.8674628Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:07.8674632Z 2025-12-04T09:40:07.8674732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8674928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8674988Z res = mod(**inputs) 2025-12-04T09:40:07.8675221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 680, in forward 2025-12-04T09:40:07.8675315Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:40:07.8675552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 358, in forward 2025-12-04T09:40:07.8675661Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:40:07.8675900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-12-04T09:40:07.8675987Z hidden_states = self.decoder(hidden_states) 2025-12-04T09:40:07.8675992Z 2025-12-04T09:40:07.8676098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:07.8676285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:07.8676346Z res = mod(**inputs) 2025-12-04T09:40:07.8676597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 685, in forward 2025-12-04T09:40:07.8676778Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:40:07.8676781Z 2025-12-04T09:40:17.4432229Z Compilation time (from dynamo_timed): 14.242845691 2025-12-04T09:40:17.4496100Z pass 2025-12-04T09:40:17.4496647Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:17.4497545Z TIMING: _recursive_pre_grad_passes:0.00554 _recursive_joint_graph_passes:0.21885 _recursive_post_grad_passes:0.06249 async_compile.wait:0.73979 code_gen:9.33157 inductor_compile:10.52203 backend_compile:12.31011 gc:0.00029 entire_frame_compile:14.24285 total_wall_time:14.24285 2025-12-04T09:40:17.4503106Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:4224 | FakeTensor.__torch_dispatch__:3268 | ProxyTorchDispatchMode.__torch_dispatch__:1259 2025-12-04T09:40:17.4503804Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-12-04T09:40:19.9288995Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:40:19.9289926Z import pynvml # type: ignore[import] 2025-12-04T09:40:23.4399911Z 2025-12-04T09:40:24.5145424Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:40:24.5146401Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:40:24.5155795Z cpu eval LayoutLMForMaskedLM 2025-12-04T09:40:25.1293394Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:25.3832377Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:25.6331142Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:33.4270207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4270731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4271069Z res = mod(**inputs) 2025-12-04T09:40:33.4271430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4272154Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4272736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4273143Z outputs = self.layoutlm( 2025-12-04T09:40:33.4273471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4273809Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4274201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4274609Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4274955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4275297Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4275692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4276090Z layer_outputs = layer_module( 2025-12-04T09:40:33.4276440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4276880Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4277339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4277743Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4278137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4278579Z self_outputs = self.self( 2025-12-04T09:40:33.4278972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4279570Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4279805Z 2025-12-04T09:40:33.4279918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4280289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4280615Z res = mod(**inputs) 2025-12-04T09:40:33.4280932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4281285Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4281683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4282084Z outputs = self.layoutlm( 2025-12-04T09:40:33.4282414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4282780Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4283174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4283681Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4284063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4284407Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4284795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4285237Z layer_outputs = layer_module( 2025-12-04T09:40:33.4285580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4285937Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4286331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4286756Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4287166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4287577Z self_outputs = self.self( 2025-12-04T09:40:33.4287957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4288565Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4288777Z 2025-12-04T09:40:33.4288897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4289263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4289582Z res = mod(**inputs) 2025-12-04T09:40:33.4289911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4290286Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4290712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4291133Z outputs = self.layoutlm( 2025-12-04T09:40:33.4291580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4291971Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4292405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4292855Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4293243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4293720Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4294138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4294538Z layer_outputs = layer_module( 2025-12-04T09:40:33.4294890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4295273Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4295707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4296161Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4296615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4297060Z self_outputs = self.self( 2025-12-04T09:40:33.4297483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4298009Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4298232Z 2025-12-04T09:40:33.4298374Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4298603Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4298886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4299278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4299626Z res = mod(**inputs) 2025-12-04T09:40:33.4299977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4300365Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4300797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4301226Z outputs = self.layoutlm( 2025-12-04T09:40:33.4301589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4301998Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4302436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4302869Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4303245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4303634Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4304034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4304436Z layer_outputs = layer_module( 2025-12-04T09:40:33.4304773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4305125Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4305505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4305902Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4306295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4306727Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4307164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4307563Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4307697Z 2025-12-04T09:40:33.4307806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4308147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4308460Z res = mod(**inputs) 2025-12-04T09:40:33.4308775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4309117Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4309501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4309890Z outputs = self.layoutlm( 2025-12-04T09:40:33.4310212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4310542Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4310927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4311313Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4311645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4311976Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4312393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4312841Z layer_outputs = layer_module( 2025-12-04T09:40:33.4313171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4313525Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4313919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4314321Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4314707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4315090Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4315537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4316007Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4316443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4316848Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4316982Z 2025-12-04T09:40:33.4317090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4317444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4317760Z res = mod(**inputs) 2025-12-04T09:40:33.4318075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4318422Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4318811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4319213Z outputs = self.layoutlm( 2025-12-04T09:40:33.4319553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4319907Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4320298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4320702Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4321046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4321387Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4321783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4322184Z layer_outputs = layer_module( 2025-12-04T09:40:33.4322559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4322918Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4323326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4323737Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4324140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4324527Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4324957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4325434Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4325873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4326353Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4326756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4327108Z return self.act(input) 2025-12-04T09:40:33.4327219Z 2025-12-04T09:40:33.4327323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4327685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4328008Z res = mod(**inputs) 2025-12-04T09:40:33.4328330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4328678Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4329074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4329495Z outputs = self.layoutlm( 2025-12-04T09:40:33.4329822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4330178Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4330575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4330979Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4331483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4331876Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4332322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4332735Z layer_outputs = layer_module( 2025-12-04T09:40:33.4333104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4333472Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4333884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4334293Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4334696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4335092Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4335520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4336002Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4336460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4336876Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4337012Z 2025-12-04T09:40:33.4337125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4337480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4337807Z res = mod(**inputs) 2025-12-04T09:40:33.4338129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4338476Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4338876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4339275Z outputs = self.layoutlm( 2025-12-04T09:40:33.4339606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4339953Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4340387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4340808Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4341144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4341495Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4341890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4342285Z layer_outputs = layer_module( 2025-12-04T09:40:33.4342625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4342991Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4343389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4343813Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4344219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4344613Z self_outputs = self.self( 2025-12-04T09:40:33.4345004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4345452Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4345653Z 2025-12-04T09:40:33.4345755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4346411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4346728Z res = mod(**inputs) 2025-12-04T09:40:33.4347042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4347399Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4347801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4348197Z outputs = self.layoutlm( 2025-12-04T09:40:33.4348519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4348865Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4349256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4349647Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4349988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4350336Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4350732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4351130Z layer_outputs = layer_module( 2025-12-04T09:40:33.4351476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4351838Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4352238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4352633Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4353023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4353409Z self_outputs = self.self( 2025-12-04T09:40:33.4353795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4354242Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4354425Z 2025-12-04T09:40:33.4354568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4354938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4355241Z res = mod(**inputs) 2025-12-04T09:40:33.4355555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4355900Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4356281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4356666Z outputs = self.layoutlm( 2025-12-04T09:40:33.4356990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4357354Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4357736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4358127Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4358463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4358800Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4359187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4359574Z layer_outputs = layer_module( 2025-12-04T09:40:33.4359913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4360260Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4360655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4361057Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4361453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4361836Z self_outputs = self.self( 2025-12-04T09:40:33.4362213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4362673Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4362866Z 2025-12-04T09:40:33.4362945Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4363152Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4363381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4363728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4364036Z res = mod(**inputs) 2025-12-04T09:40:33.4364357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4364710Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4365105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4365499Z outputs = self.layoutlm( 2025-12-04T09:40:33.4365830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4366180Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4366570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4366969Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4367312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4367658Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4368104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4368525Z layer_outputs = layer_module( 2025-12-04T09:40:33.4368876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4369236Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4369643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4370059Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4370469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4370929Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4371484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4371916Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4372062Z 2025-12-04T09:40:33.4372183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4372833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4373192Z res = mod(**inputs) 2025-12-04T09:40:33.4373539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4373916Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4374343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4374752Z outputs = self.layoutlm( 2025-12-04T09:40:33.4375089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4375444Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4375849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4376257Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4376601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4376958Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4377353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4377757Z layer_outputs = layer_module( 2025-12-04T09:40:33.4378098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4378467Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4378877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4379287Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4379691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4380092Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4380521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4380994Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4381449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4381856Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4381995Z 2025-12-04T09:40:33.4382104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4382538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4382902Z res = mod(**inputs) 2025-12-04T09:40:33.4383233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4383586Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4383978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4384365Z outputs = self.layoutlm( 2025-12-04T09:40:33.4384692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4385035Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4385429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4385844Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4386170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4386506Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4386886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4387268Z layer_outputs = layer_module( 2025-12-04T09:40:33.4387594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4387945Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4388331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4388726Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4389106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4389486Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4389900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4390349Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4390776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4391199Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4391565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4391888Z return self.act(input) 2025-12-04T09:40:33.4392003Z 2025-12-04T09:40:33.4392104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4392453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4392768Z res = mod(**inputs) 2025-12-04T09:40:33.4393081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4393422Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4393803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4394179Z outputs = self.layoutlm( 2025-12-04T09:40:33.4394502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4394840Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4395230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4395619Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4395964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4396344Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4396734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4397114Z layer_outputs = layer_module( 2025-12-04T09:40:33.4397444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4397793Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4398174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4398569Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4398951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4399341Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4399743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4400206Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4400637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4401016Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4401149Z 2025-12-04T09:40:33.4401247Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4401587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4401890Z res = mod(**inputs) 2025-12-04T09:40:33.4402187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4402535Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4402903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4403270Z outputs = self.layoutlm( 2025-12-04T09:40:33.4403572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4403895Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4404259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4404618Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4404936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4405261Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4405630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4405995Z layer_outputs = layer_module( 2025-12-04T09:40:33.4406321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4406660Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4407024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4407403Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4407784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4408161Z self_outputs = self.self( 2025-12-04T09:40:33.4408519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4408968Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4409164Z 2025-12-04T09:40:33.4410079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4410463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4410764Z res = mod(**inputs) 2025-12-04T09:40:33.4411076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4411492Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4411868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4412293Z outputs = self.layoutlm( 2025-12-04T09:40:33.4412698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4413088Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4413493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4413897Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4414243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4414608Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4414988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4415374Z layer_outputs = layer_module( 2025-12-04T09:40:33.4415714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4416058Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4416449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4416850Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4417258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4417659Z self_outputs = self.self( 2025-12-04T09:40:33.4418032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4418476Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4418660Z 2025-12-04T09:40:33.4418769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4419112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4419431Z res = mod(**inputs) 2025-12-04T09:40:33.4419746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4420087Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4420476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4420864Z outputs = self.layoutlm( 2025-12-04T09:40:33.4421187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4421522Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4421908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4422302Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4422641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4422997Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4423388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4423860Z layer_outputs = layer_module( 2025-12-04T09:40:33.4424191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4424557Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4424940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4425319Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4425703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4426076Z self_outputs = self.self( 2025-12-04T09:40:33.4426439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4426902Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4427101Z 2025-12-04T09:40:33.4427181Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4427387Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4427611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4427942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4428244Z res = mod(**inputs) 2025-12-04T09:40:33.4428552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4428881Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4429261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4429639Z outputs = self.layoutlm( 2025-12-04T09:40:33.4429957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4430286Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4430673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4431066Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4431396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4431734Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4432122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4432508Z layer_outputs = layer_module( 2025-12-04T09:40:33.4432840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4433196Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4433589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4433989Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4434377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4434828Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4435257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4435647Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4435788Z 2025-12-04T09:40:33.4435887Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4436236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4436552Z res = mod(**inputs) 2025-12-04T09:40:33.4436858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4437249Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4437655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4438035Z outputs = self.layoutlm( 2025-12-04T09:40:33.4438370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4438705Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4439086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4439467Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4439803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4440166Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4440556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4440943Z layer_outputs = layer_module( 2025-12-04T09:40:33.4441286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4441642Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4442030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4442437Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4442832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4443216Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4443634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4444105Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4444541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4444941Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4445073Z 2025-12-04T09:40:33.4445174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4445526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4445840Z res = mod(**inputs) 2025-12-04T09:40:33.4446147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4446492Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4446894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4447293Z outputs = self.layoutlm( 2025-12-04T09:40:33.4447622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4447982Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4448385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4448785Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4449132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4449485Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4449886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4450280Z layer_outputs = layer_module( 2025-12-04T09:40:33.4450664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4451032Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4451550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4451990Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4452415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4452830Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4453275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4453776Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4454235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4454663Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4455030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4455370Z return self.act(input) 2025-12-04T09:40:33.4455477Z 2025-12-04T09:40:33.4455585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4455931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4456246Z res = mod(**inputs) 2025-12-04T09:40:33.4456560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4456901Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4457283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4457673Z outputs = self.layoutlm( 2025-12-04T09:40:33.4458001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4458346Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4458727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4459115Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4459450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4459785Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4460173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4460561Z layer_outputs = layer_module( 2025-12-04T09:40:33.4460903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4461253Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4461649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4462046Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4462427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4462814Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4463234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4463711Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4464146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4464548Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4464686Z 2025-12-04T09:40:33.4464825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4465195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4465503Z res = mod(**inputs) 2025-12-04T09:40:33.4465823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4466167Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4466550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4466937Z outputs = self.layoutlm( 2025-12-04T09:40:33.4467261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4467636Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4468016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4468410Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4468751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4469130Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4469507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4469891Z layer_outputs = layer_module( 2025-12-04T09:40:33.4470224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4470569Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4470959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4471356Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4471749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4472180Z self_outputs = self.self( 2025-12-04T09:40:33.4472680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4473150Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4473343Z 2025-12-04T09:40:33.4473455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4473801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4474116Z res = mod(**inputs) 2025-12-04T09:40:33.4474432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4474781Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4475165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4475546Z outputs = self.layoutlm( 2025-12-04T09:40:33.4475867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4476197Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4476578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4476964Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4477286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4477628Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4478021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4478483Z layer_outputs = layer_module( 2025-12-04T09:40:33.4478833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4479178Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4479561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4479949Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4480325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4480701Z self_outputs = self.self( 2025-12-04T09:40:33.4481065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4481557Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4481746Z 2025-12-04T09:40:33.4481850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4482190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4482502Z res = mod(**inputs) 2025-12-04T09:40:33.4482810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4483157Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4483555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4483924Z outputs = self.layoutlm( 2025-12-04T09:40:33.4484245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4484589Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4484977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4485367Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4485699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4486038Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4486421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4486803Z layer_outputs = layer_module( 2025-12-04T09:40:33.4487145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4487507Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4487904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4488315Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4488717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4489111Z self_outputs = self.self( 2025-12-04T09:40:33.4489488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4489959Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4490154Z 2025-12-04T09:40:33.4490241Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4490452Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4490678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4491038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4491456Z res = mod(**inputs) 2025-12-04T09:40:33.4491850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4492241Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4492699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4493098Z outputs = self.layoutlm( 2025-12-04T09:40:33.4493424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4493778Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4494177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4494568Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4494910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4495287Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4495675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4496055Z layer_outputs = layer_module( 2025-12-04T09:40:33.4496393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4496746Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4497126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4497523Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4497912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4498354Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4498791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4499196Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4499326Z 2025-12-04T09:40:33.4499433Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4499783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4500094Z res = mod(**inputs) 2025-12-04T09:40:33.4500411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4500753Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4501131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4501513Z outputs = self.layoutlm( 2025-12-04T09:40:33.4501840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4502182Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4502564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4502958Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4503297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4503621Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4503994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4504368Z layer_outputs = layer_module( 2025-12-04T09:40:33.4504690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4505028Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4505450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4505864Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4506252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4506624Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4507035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4507489Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4507909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4508303Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4508459Z 2025-12-04T09:40:33.4508556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4508903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4509208Z res = mod(**inputs) 2025-12-04T09:40:33.4509516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4509850Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4510232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4510610Z outputs = self.layoutlm( 2025-12-04T09:40:33.4510926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4511259Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4511627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4512011Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4512343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4512678Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4513047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4513428Z layer_outputs = layer_module( 2025-12-04T09:40:33.4513757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4514096Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4514479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4514871Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4515253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4515630Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4516023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4516461Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4516869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4517266Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4517616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4517935Z return self.act(input) 2025-12-04T09:40:33.4518037Z 2025-12-04T09:40:33.4518134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4518465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4518801Z res = mod(**inputs) 2025-12-04T09:40:33.4519134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4519453Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4519827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4520200Z outputs = self.layoutlm( 2025-12-04T09:40:33.4520501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4520828Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4521198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4521597Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4521914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4522239Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4522606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4522976Z layer_outputs = layer_module( 2025-12-04T09:40:33.4523290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4523630Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4524010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4524381Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4524752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4525120Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4525526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4525983Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4526419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4526808Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4526938Z 2025-12-04T09:40:33.4527041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4527376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4527686Z res = mod(**inputs) 2025-12-04T09:40:33.4527996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4528331Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4528716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4529095Z outputs = self.layoutlm( 2025-12-04T09:40:33.4529412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4529737Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4530116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4530497Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4530827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4531170Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4531634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4532068Z layer_outputs = layer_module( 2025-12-04T09:40:33.4532421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4532812Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4533248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4533683Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4534108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4534493Z self_outputs = self.self( 2025-12-04T09:40:33.4534869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4535338Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4535541Z 2025-12-04T09:40:33.4535641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4535991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4536306Z res = mod(**inputs) 2025-12-04T09:40:33.4536612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4536957Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4537344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4537729Z outputs = self.layoutlm( 2025-12-04T09:40:33.4538044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4538383Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4538768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4539151Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4539488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4539828Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4540212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4540591Z layer_outputs = layer_module( 2025-12-04T09:40:33.4540929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4541280Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4541664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4542062Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4542457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4542839Z self_outputs = self.self( 2025-12-04T09:40:33.4543205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4543652Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4543843Z 2025-12-04T09:40:33.4543946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4544294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4544600Z res = mod(**inputs) 2025-12-04T09:40:33.4544918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4545265Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4545682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4546088Z outputs = self.layoutlm( 2025-12-04T09:40:33.4546415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4546737Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4547094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4547462Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4547783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4548101Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4548485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4548858Z layer_outputs = layer_module( 2025-12-04T09:40:33.4549180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4549507Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4549878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4550256Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4550629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4550988Z self_outputs = self.self( 2025-12-04T09:40:33.4551345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4551778Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4551961Z 2025-12-04T09:40:33.4552038Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4552237Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4552454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4552782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4553069Z res = mod(**inputs) 2025-12-04T09:40:33.4553367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4553690Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4554048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4554414Z outputs = self.layoutlm( 2025-12-04T09:40:33.4554718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4555042Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4555400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4555767Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4556083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4556405Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4556760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4557129Z layer_outputs = layer_module( 2025-12-04T09:40:33.4557452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4557780Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4558191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4558583Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4558956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4559368Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4559784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4560163Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4560289Z 2025-12-04T09:40:33.4560391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4560713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4561029Z res = mod(**inputs) 2025-12-04T09:40:33.4561335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4561658Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4562030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4562400Z outputs = self.layoutlm( 2025-12-04T09:40:33.4562706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4563026Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4563394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4563765Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4564079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4564407Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4564775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4565148Z layer_outputs = layer_module( 2025-12-04T09:40:33.4565461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4565798Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4566171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4566549Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4566915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4567278Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4567681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4568116Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4568530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4568910Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4569036Z 2025-12-04T09:40:33.4569138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4569463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4569759Z res = mod(**inputs) 2025-12-04T09:40:33.4570055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4570371Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4570742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4571158Z outputs = self.layoutlm( 2025-12-04T09:40:33.4571556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4571959Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4572541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4572979Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4573395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4573744Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4574146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4574595Z layer_outputs = layer_module( 2025-12-04T09:40:33.4574940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4575317Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4575726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4576139Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4576539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4576937Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4577374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4577857Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4578300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4578764Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4579153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4579494Z return self.act(input) 2025-12-04T09:40:33.4579612Z 2025-12-04T09:40:33.4579717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4580072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4580401Z res = mod(**inputs) 2025-12-04T09:40:33.4580718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4581074Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4581470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4581864Z outputs = self.layoutlm( 2025-12-04T09:40:33.4582199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4582615Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4583009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4583404Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4583747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4584100Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4584497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4584897Z layer_outputs = layer_module( 2025-12-04T09:40:33.4585227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4585643Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4586055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4586447Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4586826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4587197Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4587594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4588060Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4588495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4588902Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4589036Z 2025-12-04T09:40:33.4589138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4589476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4589779Z res = mod(**inputs) 2025-12-04T09:40:33.4590076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4590408Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4590780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4591157Z outputs = self.layoutlm( 2025-12-04T09:40:33.4591468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4591804Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4592181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4592554Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4592883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4593232Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4593605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4593973Z layer_outputs = layer_module( 2025-12-04T09:40:33.4594303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4594648Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4595024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4595413Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4595800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4596182Z self_outputs = self.self( 2025-12-04T09:40:33.4596540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4596987Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4597181Z 2025-12-04T09:40:33.4597278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4597615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4597909Z res = mod(**inputs) 2025-12-04T09:40:33.4598212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4598553Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4598968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4599363Z outputs = self.layoutlm( 2025-12-04T09:40:33.4599680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4600013Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4600382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4600764Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4601093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4601423Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4601815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4602201Z layer_outputs = layer_module( 2025-12-04T09:40:33.4602537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4602873Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4603258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4603646Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4604031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4604402Z self_outputs = self.self( 2025-12-04T09:40:33.4604771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4605211Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4605391Z 2025-12-04T09:40:33.4605497Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4605831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4606136Z res = mod(**inputs) 2025-12-04T09:40:33.4606444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4606774Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4607152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4607530Z outputs = self.layoutlm( 2025-12-04T09:40:33.4607848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4608177Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4608562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4608956Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4609283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4609631Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4610023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4610411Z layer_outputs = layer_module( 2025-12-04T09:40:33.4610744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4611102Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4611565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4612026Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4612504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4612959Z self_outputs = self.self( 2025-12-04T09:40:33.4613388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4613880Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4614085Z 2025-12-04T09:40:33.4614163Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4614372Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4614599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4614941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4615276Z res = mod(**inputs) 2025-12-04T09:40:33.4615597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4615937Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4616325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4616711Z outputs = self.layoutlm( 2025-12-04T09:40:33.4617040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4617388Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4617801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4618194Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4618527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4618870Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4619263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4619655Z layer_outputs = layer_module( 2025-12-04T09:40:33.4619995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4620353Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4620747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4621145Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4621538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4621983Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4622428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4622825Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4622965Z 2025-12-04T09:40:33.4623066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4623412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4623728Z res = mod(**inputs) 2025-12-04T09:40:33.4624034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4624380Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4624770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4625156Z outputs = self.layoutlm( 2025-12-04T09:40:33.4625477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4625865Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4626272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4626660Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4626994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4627332Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4627715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4628096Z layer_outputs = layer_module( 2025-12-04T09:40:33.4628432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4628803Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4629192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4629598Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4629991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4630375Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4630794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4631250Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4631675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4632062Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4632193Z 2025-12-04T09:40:33.4632291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4632636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4632949Z res = mod(**inputs) 2025-12-04T09:40:33.4633249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4633584Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4633962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4634345Z outputs = self.layoutlm( 2025-12-04T09:40:33.4634652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4634988Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4635363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4635744Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4636074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4636406Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4636782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4637159Z layer_outputs = layer_module( 2025-12-04T09:40:33.4637489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4637836Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4638221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4638612Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4639031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4639419Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4639817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4640271Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4640688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4641098Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4641446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4641771Z return self.act(input) 2025-12-04T09:40:33.4641900Z 2025-12-04T09:40:33.4642006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4642351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4642649Z res = mod(**inputs) 2025-12-04T09:40:33.4642955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4643293Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4643664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4644044Z outputs = self.layoutlm( 2025-12-04T09:40:33.4644359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4644695Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4645065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4645452Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4645785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4646110Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4646493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4646872Z layer_outputs = layer_module( 2025-12-04T09:40:33.4647213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4647562Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4647961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4648363Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4648761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4649141Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4649555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4650028Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4650467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4650554Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4650558Z 2025-12-04T09:40:33.4650657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4650856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4650917Z res = mod(**inputs) 2025-12-04T09:40:33.4651132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4651245Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4651576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4651684Z outputs = self.layoutlm( 2025-12-04T09:40:33.4651900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4651973Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4652247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4652320Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4652535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4652633Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4652901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4652985Z layer_outputs = layer_module( 2025-12-04T09:40:33.4653210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4653289Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4653561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4653643Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4653919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4653997Z self_outputs = self.self( 2025-12-04T09:40:33.4654263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4654420Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4654424Z 2025-12-04T09:40:33.4654533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4654730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4654803Z res = mod(**inputs) 2025-12-04T09:40:33.4655018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4655097Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4655364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4655434Z outputs = self.layoutlm( 2025-12-04T09:40:33.4655653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4655727Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4655995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4656075Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4656288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4656365Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4656630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4656702Z layer_outputs = layer_module( 2025-12-04T09:40:33.4656926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4657004Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4657274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4657390Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4657673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4657748Z self_outputs = self.self( 2025-12-04T09:40:33.4658011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4658145Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4658156Z 2025-12-04T09:40:33.4658259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4658457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4658525Z res = mod(**inputs) 2025-12-04T09:40:33.4658768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4658838Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4659116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4659184Z outputs = self.layoutlm( 2025-12-04T09:40:33.4659405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4659476Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4659740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4659816Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4660045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4660115Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4660385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4660455Z layer_outputs = layer_module( 2025-12-04T09:40:33.4660677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4660753Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4661011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4661097Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4661359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4661428Z self_outputs = self.self( 2025-12-04T09:40:33.4661701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4661844Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4661850Z 2025-12-04T09:40:33.4661939Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4662016Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4662117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4662324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4662386Z res = mod(**inputs) 2025-12-04T09:40:33.4662609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4662682Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4662948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4663026Z outputs = self.layoutlm( 2025-12-04T09:40:33.4663248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4663350Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4663637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4663707Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4663923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4663993Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4664249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4664324Z layer_outputs = layer_module( 2025-12-04T09:40:33.4664536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4664634Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4664905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4664990Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4665255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4665380Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4665648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4665741Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4665744Z 2025-12-04T09:40:33.4665845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4666049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4666113Z res = mod(**inputs) 2025-12-04T09:40:33.4666332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4666413Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4666678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4666744Z outputs = self.layoutlm( 2025-12-04T09:40:33.4666965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4667037Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4667310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4667381Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4667598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4667677Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4667947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4668027Z layer_outputs = layer_module( 2025-12-04T09:40:33.4668249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4668326Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4668599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4668684Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4668944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4669028Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4669362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4669536Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4669807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4669885Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4669889Z 2025-12-04T09:40:33.4669997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4670196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4670267Z res = mod(**inputs) 2025-12-04T09:40:33.4670493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4670588Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4670877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4670954Z outputs = self.layoutlm( 2025-12-04T09:40:33.4671168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4671247Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4671512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4671588Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4671803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4671871Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4672144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4672215Z layer_outputs = layer_module( 2025-12-04T09:40:33.4672573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4672656Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4672923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4673013Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4673272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4673347Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4673651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4673769Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4674045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4674160Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4674372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4674450Z return self.act(input) 2025-12-04T09:40:33.4674454Z 2025-12-04T09:40:33.4674555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4674758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4674819Z res = mod(**inputs) 2025-12-04T09:40:33.4675035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4675114Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4675380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4675451Z outputs = self.layoutlm( 2025-12-04T09:40:33.4675739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4675836Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4676108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4676181Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4676394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4676472Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4676736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4676840Z layer_outputs = layer_module( 2025-12-04T09:40:33.4677076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4677158Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4677439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4677523Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4677784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4677869Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4678169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4678309Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4678581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4678662Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4678667Z 2025-12-04T09:40:33.4678780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4678980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4679042Z res = mod(**inputs) 2025-12-04T09:40:33.4679268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4679342Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4679617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4679685Z outputs = self.layoutlm( 2025-12-04T09:40:33.4679903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4679983Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4680255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4680335Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4680552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4680623Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4680897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4680964Z layer_outputs = layer_module( 2025-12-04T09:40:33.4681188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4681276Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4681548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4681670Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4681958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4682023Z self_outputs = self.self( 2025-12-04T09:40:33.4682283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4682416Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4682420Z 2025-12-04T09:40:33.4682522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4682709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4682767Z res = mod(**inputs) 2025-12-04T09:40:33.4682997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4683067Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4683325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4683396Z outputs = self.layoutlm( 2025-12-04T09:40:33.4683597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4683671Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4683920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4683987Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4684198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4684265Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4684516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4684590Z layer_outputs = layer_module( 2025-12-04T09:40:33.4684795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4684877Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4685126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4685200Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4685461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4685526Z self_outputs = self.self( 2025-12-04T09:40:33.4685783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4685914Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4685919Z 2025-12-04T09:40:33.4686014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4686205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4686264Z res = mod(**inputs) 2025-12-04T09:40:33.4686466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4686539Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4686790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4686858Z outputs = self.layoutlm( 2025-12-04T09:40:33.4687059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4687128Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4687418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4687503Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4687715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4687783Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4688035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4688109Z layer_outputs = layer_module( 2025-12-04T09:40:33.4688319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4688391Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4688674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4688751Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4689010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4689074Z self_outputs = self.self( 2025-12-04T09:40:33.4689327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4689469Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4689473Z 2025-12-04T09:40:33.4689549Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4689629Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4689725Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4689912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4689979Z res = mod(**inputs) 2025-12-04T09:40:33.4690187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4690256Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4690518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4690582Z outputs = self.layoutlm( 2025-12-04T09:40:33.4690793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4690860Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4691113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4691187Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4691478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4691564Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4691865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4691942Z layer_outputs = layer_module( 2025-12-04T09:40:33.4692187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4692271Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4692572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4692662Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4692941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4693086Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4693417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4693519Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4693523Z 2025-12-04T09:40:33.4693633Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4693832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4693894Z res = mod(**inputs) 2025-12-04T09:40:33.4694127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4694197Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4694462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4694549Z outputs = self.layoutlm( 2025-12-04T09:40:33.4694761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4694840Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4695101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4695170Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4695389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4695458Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4695727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4695793Z layer_outputs = layer_module( 2025-12-04T09:40:33.4696009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4696093Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4696355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4696446Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4696705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4696781Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4697090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4697209Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4697483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4697571Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4697575Z 2025-12-04T09:40:33.4697674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4697879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4697942Z res = mod(**inputs) 2025-12-04T09:40:33.4698156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4698232Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4698493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4698568Z outputs = self.layoutlm( 2025-12-04T09:40:33.4698782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4698853Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4699125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4699229Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4699455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4699533Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4699797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4699871Z layer_outputs = layer_module( 2025-12-04T09:40:33.4700091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4700169Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4700440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4700544Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4700812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4700889Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4701186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4701311Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4701577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4701686Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4701904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4701971Z return self.act(input) 2025-12-04T09:40:33.4701976Z 2025-12-04T09:40:33.4702083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4702284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4702347Z res = mod(**inputs) 2025-12-04T09:40:33.4702570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4702642Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4702913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4702981Z outputs = self.layoutlm( 2025-12-04T09:40:33.4703196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4703273Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4703537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4703609Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4703830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4703901Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4704173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4704242Z layer_outputs = layer_module( 2025-12-04T09:40:33.4704460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4704556Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4704816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4704896Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4705154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4705265Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4705579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4705711Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4705979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4706072Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4706075Z 2025-12-04T09:40:33.4706179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4706383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4706467Z res = mod(**inputs) 2025-12-04T09:40:33.4706678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4706761Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4707024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4707091Z outputs = self.layoutlm( 2025-12-04T09:40:33.4707309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4707380Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4707645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4707714Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4707923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4708001Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4708262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4708337Z layer_outputs = layer_module( 2025-12-04T09:40:33.4708552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4708629Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4708897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4708975Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4709234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4709307Z self_outputs = self.self( 2025-12-04T09:40:33.4709566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4709720Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4709725Z 2025-12-04T09:40:33.4709825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4710013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4710079Z res = mod(**inputs) 2025-12-04T09:40:33.4710290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4710368Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4710624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4710690Z outputs = self.layoutlm( 2025-12-04T09:40:33.4710910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4710981Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4711273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4711367Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4711575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4711650Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4711909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4711977Z layer_outputs = layer_module( 2025-12-04T09:40:33.4712197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4712271Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4712551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4712641Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4712903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4712977Z self_outputs = self.self( 2025-12-04T09:40:33.4713234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4713366Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4713370Z 2025-12-04T09:40:33.4713479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4713669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4713736Z res = mod(**inputs) 2025-12-04T09:40:33.4713950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4714022Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4714285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4714350Z outputs = self.layoutlm( 2025-12-04T09:40:33.4714558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4714632Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4714889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4714964Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4715173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4715243Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4715511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4715577Z layer_outputs = layer_module( 2025-12-04T09:40:33.4715796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4715871Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4716128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4716210Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4716465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4716541Z self_outputs = self.self( 2025-12-04T09:40:33.4716795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4716958Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4716978Z 2025-12-04T09:40:33.4717062Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4717136Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4717231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4717424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4717485Z res = mod(**inputs) 2025-12-04T09:40:33.4717688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4717762Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4718015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4718106Z outputs = self.layoutlm( 2025-12-04T09:40:33.4718313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4718379Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4718639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4718704Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4718916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4718981Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4719231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4719304Z layer_outputs = layer_module( 2025-12-04T09:40:33.4719514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4719590Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4719856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4719935Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4720199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4720323Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4720578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4720663Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4720666Z 2025-12-04T09:40:33.4720765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4720964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4721027Z res = mod(**inputs) 2025-12-04T09:40:33.4721239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4721318Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4721574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4721639Z outputs = self.layoutlm( 2025-12-04T09:40:33.4721855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4721923Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4722191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4722260Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4722472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4722581Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4722842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4722929Z layer_outputs = layer_module( 2025-12-04T09:40:33.4723151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4723226Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4723490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4723571Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4723828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4723928Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4724214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4724334Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4724587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4724663Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4724666Z 2025-12-04T09:40:33.4724766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4724950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4725016Z res = mod(**inputs) 2025-12-04T09:40:33.4725221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4725290Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4725559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4725626Z outputs = self.layoutlm( 2025-12-04T09:40:33.4725841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4725917Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4726175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4726250Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4726461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4726528Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4726792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4726861Z layer_outputs = layer_module( 2025-12-04T09:40:33.4727077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4727160Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4727419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4727507Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4727763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4727834Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4728131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4728247Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4728557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4728682Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4728891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4728965Z return self.act(input) 2025-12-04T09:40:33.4728969Z 2025-12-04T09:40:33.4729071Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4729269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4729338Z res = mod(**inputs) 2025-12-04T09:40:33.4729553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4729632Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4729917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4729987Z outputs = self.layoutlm( 2025-12-04T09:40:33.4730216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4730287Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4730557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4730627Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4730844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4730922Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4731185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4731255Z layer_outputs = layer_module( 2025-12-04T09:40:33.4731556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4731639Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4731912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4731992Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4732250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4732334Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4732628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4732773Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4733055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4733140Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4733146Z 2025-12-04T09:40:33.4733263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4733469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4733535Z res = mod(**inputs) 2025-12-04T09:40:33.4733770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4733845Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4734139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4734207Z outputs = self.layoutlm( 2025-12-04T09:40:33.4734420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4734498Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4734795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4734883Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4735112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4735183Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4735462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4735530Z layer_outputs = layer_module( 2025-12-04T09:40:33.4735754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4735840Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4736125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4736218Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4736485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4736554Z self_outputs = self.self( 2025-12-04T09:40:33.4736824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4736966Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4736970Z 2025-12-04T09:40:33.4737080Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4737274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4737339Z res = mod(**inputs) 2025-12-04T09:40:33.4737562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4737637Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4737906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4737981Z outputs = self.layoutlm( 2025-12-04T09:40:33.4738199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4738277Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4738543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4738613Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4738838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4738910Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4739178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4739259Z layer_outputs = layer_module( 2025-12-04T09:40:33.4739477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4739559Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4739824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4739903Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4740176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4740244Z self_outputs = self.self( 2025-12-04T09:40:33.4740518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4740692Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4740712Z 2025-12-04T09:40:33.4740816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4741022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4741084Z res = mod(**inputs) 2025-12-04T09:40:33.4741304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4741382Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4741646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4741720Z outputs = self.layoutlm( 2025-12-04T09:40:33.4741935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4742026Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4742302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4742374Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4742590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4742669Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4742930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4743005Z layer_outputs = layer_module( 2025-12-04T09:40:33.4743223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4743300Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4743572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4743654Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4743925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4743992Z self_outputs = self.self( 2025-12-04T09:40:33.4744264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4744412Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4744416Z 2025-12-04T09:40:33.4744494Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4744572Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4744680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4744875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4744943Z res = mod(**inputs) 2025-12-04T09:40:33.4745158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4745230Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4745563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4745629Z outputs = self.layoutlm( 2025-12-04T09:40:33.4745843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4745919Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4746185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4746259Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4746475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4746574Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4746865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4746934Z layer_outputs = layer_module( 2025-12-04T09:40:33.4747162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4747240Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4747505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4747593Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4747932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4748092Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4748364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4748459Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4748463Z 2025-12-04T09:40:33.4748567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4748756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4748817Z res = mod(**inputs) 2025-12-04T09:40:33.4749033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4749102Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4749365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4749431Z outputs = self.layoutlm( 2025-12-04T09:40:33.4749642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4749721Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4749983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4750052Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4750270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4750339Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4750607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4750675Z layer_outputs = layer_module( 2025-12-04T09:40:33.4750886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4750974Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4751245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4751336Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4751593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4751665Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4751967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4752084Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4752350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4752439Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4752443Z 2025-12-04T09:40:33.4752577Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4752806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4752868Z res = mod(**inputs) 2025-12-04T09:40:33.4753082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4753160Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4753423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4753496Z outputs = self.layoutlm( 2025-12-04T09:40:33.4753706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4753774Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4754053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4754125Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4754337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4754413Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4754670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4754745Z layer_outputs = layer_module( 2025-12-04T09:40:33.4754958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4755031Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4755293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4755375Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4755628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4755709Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4756005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4756131Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4756406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4756513Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4756731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4756801Z return self.act(input) 2025-12-04T09:40:33.4756804Z 2025-12-04T09:40:33.4756910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4757110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4757175Z res = mod(**inputs) 2025-12-04T09:40:33.4757399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4757472Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4757736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4757811Z outputs = self.layoutlm( 2025-12-04T09:40:33.4758037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4758114Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4758385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4758454Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4758702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4758789Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4759056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4759125Z layer_outputs = layer_module( 2025-12-04T09:40:33.4759339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4759421Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4759679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4759779Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4760039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4760112Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4760408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4760535Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4760792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4760877Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4760881Z 2025-12-04T09:40:33.4760980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4761180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4761242Z res = mod(**inputs) 2025-12-04T09:40:33.4761453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4761531Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4761790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4761857Z outputs = self.layoutlm( 2025-12-04T09:40:33.4762074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4762143Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4762408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4762476Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4762682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4762758Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4763018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4763088Z layer_outputs = layer_module( 2025-12-04T09:40:33.4763310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4763387Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4763653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4763732Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4763997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4764071Z self_outputs = self.self( 2025-12-04T09:40:33.4764330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4764536Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4764556Z 2025-12-04T09:40:33.4764659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4764849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4764920Z res = mod(**inputs) 2025-12-04T09:40:33.4765131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4765201Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4765470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4765538Z outputs = self.layoutlm( 2025-12-04T09:40:33.4765775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4765844Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4766112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4766190Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4766394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4766471Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4766727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4766792Z layer_outputs = layer_module( 2025-12-04T09:40:33.4767011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4767086Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4767341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4767425Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4767675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4767748Z self_outputs = self.self( 2025-12-04T09:40:33.4767999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4768127Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4768131Z 2025-12-04T09:40:33.4768235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4768419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4768487Z res = mod(**inputs) 2025-12-04T09:40:33.4768693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4768761Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4769021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4769085Z outputs = self.layoutlm( 2025-12-04T09:40:33.4769288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4769362Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4769616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4769690Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4769892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4769959Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4770253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4770349Z layer_outputs = layer_module( 2025-12-04T09:40:33.4770560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4770644Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4770904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4770991Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4771249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4771383Z self_outputs = self.self( 2025-12-04T09:40:33.4771716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4771873Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4771880Z 2025-12-04T09:40:33.4771976Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4772070Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4772179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4772567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4772649Z res = mod(**inputs) 2025-12-04T09:40:33.4772870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4772953Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4773224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4773304Z outputs = self.layoutlm( 2025-12-04T09:40:33.4773531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4773601Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4773863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4773929Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4774135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4774211Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4774468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4774541Z layer_outputs = layer_module( 2025-12-04T09:40:33.4774750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4774825Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4775090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4775170Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4775431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4775554Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4775809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4775894Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4775898Z 2025-12-04T09:40:33.4775994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4776185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4776243Z res = mod(**inputs) 2025-12-04T09:40:33.4776518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4776617Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4776868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4776932Z outputs = self.layoutlm( 2025-12-04T09:40:33.4777140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4777205Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4777465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4777532Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4777760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4777839Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4778093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4778161Z layer_outputs = layer_module( 2025-12-04T09:40:33.4778375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4778447Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4778705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4778785Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4779032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4779114Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4779397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4779521Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4779769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4779845Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4779848Z 2025-12-04T09:40:33.4779951Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4780135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4780194Z res = mod(**inputs) 2025-12-04T09:40:33.4780406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4780476Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4780735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4780800Z outputs = self.layoutlm( 2025-12-04T09:40:33.4781003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4781080Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4781331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4781404Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4781606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4781673Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4781929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4781997Z layer_outputs = layer_module( 2025-12-04T09:40:33.4782231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4782328Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4782584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4782671Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4782917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4782987Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4783284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4783415Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4783676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4783783Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4783989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4784061Z return self.act(input) 2025-12-04T09:40:33.4784065Z 2025-12-04T09:40:33.4784162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4784348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4784415Z res = mod(**inputs) 2025-12-04T09:40:33.4784619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4784692Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4784950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4785014Z outputs = self.layoutlm( 2025-12-04T09:40:33.4785230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4785299Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4785554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4785630Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4785835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4785908Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4786165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4786233Z layer_outputs = layer_module( 2025-12-04T09:40:33.4786454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4786529Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4786793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4786872Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4787116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4787191Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4787474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4787599Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4787893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4787986Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4787989Z 2025-12-04T09:40:33.4788090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4788274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4788333Z res = mod(**inputs) 2025-12-04T09:40:33.4788544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4788611Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4788868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4788932Z outputs = self.layoutlm( 2025-12-04T09:40:33.4789154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4789231Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4789487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4789555Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4789766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4789831Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4790090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4790155Z layer_outputs = layer_module( 2025-12-04T09:40:33.4790361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4790442Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4790697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4790775Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4791034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4791098Z self_outputs = self.self( 2025-12-04T09:40:33.4791419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T09:40:33.4791549Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4791552Z 2025-12-04T09:40:33.4791643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4791830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4791889Z res = mod(**inputs) 2025-12-04T09:40:33.4792096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4792162Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4792407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4792477Z outputs = self.layoutlm( 2025-12-04T09:40:33.4792674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4792739Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4792998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4793066Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4793276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4793343Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4793632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4793722Z layer_outputs = layer_module( 2025-12-04T09:40:33.4793937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4794019Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4794265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4794340Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4794590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4794652Z self_outputs = self.self( 2025-12-04T09:40:33.4794915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T09:40:33.4795052Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4795057Z 2025-12-04T09:40:33.4795153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4795346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4795406Z res = mod(**inputs) 2025-12-04T09:40:33.4795610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4795686Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4795940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4796012Z outputs = self.layoutlm( 2025-12-04T09:40:33.4796215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4796280Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4796540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4796609Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4796813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4796888Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4797139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4797212Z layer_outputs = layer_module( 2025-12-04T09:40:33.4797417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4797490Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4797761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4797836Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4798087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T09:40:33.4798149Z self_outputs = self.self( 2025-12-04T09:40:33.4798394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T09:40:33.4798530Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T09:40:33.4798533Z 2025-12-04T09:40:33.4798607Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4798678Z cudagraph partition due to non gpu ops 2025-12-04T09:40:33.4798780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4798966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4799030Z res = mod(**inputs) 2025-12-04T09:40:33.4799261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4799344Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4799600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4799663Z outputs = self.layoutlm( 2025-12-04T09:40:33.4799863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4799936Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4800182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4800272Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4800481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4800548Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4800821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4800885Z layer_outputs = layer_module( 2025-12-04T09:40:33.4801095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4801174Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4801426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T09:40:33.4801507Z self_attention_outputs = self.attention( 2025-12-04T09:40:33.4801758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T09:40:33.4801880Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:40:33.4802142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T09:40:33.4802217Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4802221Z 2025-12-04T09:40:33.4802320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4802507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4802565Z res = mod(**inputs) 2025-12-04T09:40:33.4802773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4820721Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4821095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4821202Z outputs = self.layoutlm( 2025-12-04T09:40:33.4821442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4821521Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4821801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4821877Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4822099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4822179Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4822463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4822542Z layer_outputs = layer_module( 2025-12-04T09:40:33.4822762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4822843Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4823245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4823357Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4823618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4823693Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4823976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4824104Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4824355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T09:40:33.4824467Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4824475Z 2025-12-04T09:40:33.4824591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4824790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4824863Z res = mod(**inputs) 2025-12-04T09:40:33.4825072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4825142Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4825404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4825476Z outputs = self.layoutlm( 2025-12-04T09:40:33.4825686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4825757Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4826010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4826089Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4826298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4826364Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4826622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4826693Z layer_outputs = layer_module( 2025-12-04T09:40:33.4826915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4826995Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4827254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4827345Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4827601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4827676Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4827973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T09:40:33.4828091Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:40:33.4828357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T09:40:33.4828465Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:40:33.4828672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:40:33.4828751Z return self.act(input) 2025-12-04T09:40:33.4828755Z 2025-12-04T09:40:33.4828858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4829097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4829180Z res = mod(**inputs) 2025-12-04T09:40:33.4829395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4829470Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4829731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T09:40:33.4829799Z outputs = self.layoutlm( 2025-12-04T09:40:33.4830017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4830085Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4830369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T09:40:33.4830444Z encoder_outputs = self.encoder( 2025-12-04T09:40:33.4830654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4830733Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4830990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T09:40:33.4831067Z layer_outputs = layer_module( 2025-12-04T09:40:33.4831282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:40:33.4831361Z return super().__call__(*args, **kwargs) 2025-12-04T09:40:33.4831626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T09:40:33.4831710Z layer_output = apply_chunking_to_forward( 2025-12-04T09:40:33.4831964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:40:33.4832047Z return forward_fn(*input_tensors) 2025-12-04T09:40:33.4832334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T09:40:33.4832472Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:40:33.4832732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T09:40:33.4832812Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4832816Z 2025-12-04T09:40:33.4832926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4833121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4833195Z res = mod(**inputs) 2025-12-04T09:40:33.4833409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4833481Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4833745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 735, in forward 2025-12-04T09:40:33.4833837Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:40:33.4834118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 456, in forward 2025-12-04T09:40:33.4834234Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:40:33.4834491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 444, in forward 2025-12-04T09:40:33.4834589Z hidden_states = self.transform(hidden_states) 2025-12-04T09:40:33.4834851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 419, in forward 2025-12-04T09:40:33.4834971Z hidden_states = self.dense(hidden_states) 2025-12-04T09:40:33.4834990Z 2025-12-04T09:40:33.4835101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4835296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4835365Z res = mod(**inputs) 2025-12-04T09:40:33.4835575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4835645Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4835911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 735, in forward 2025-12-04T09:40:33.4835999Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:40:33.4836261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 456, in forward 2025-12-04T09:40:33.4836397Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:40:33.4836658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 445, in forward 2025-12-04T09:40:33.4836755Z hidden_states = self.decoder(hidden_states) 2025-12-04T09:40:33.4836759Z 2025-12-04T09:40:33.4836858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:40:33.4837050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:40:33.4837118Z res = mod(**inputs) 2025-12-04T09:40:33.4837331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:40:33.4837409Z output = func(self, *args, **kwargs) 2025-12-04T09:40:33.4837670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 740, in forward 2025-12-04T09:40:33.4837741Z masked_lm_loss = loss_fct( 2025-12-04T09:40:33.4837745Z 2025-12-04T09:40:43.4067466Z Compilation time (from dynamo_timed): 17.093680167 2025-12-04T09:40:43.4134796Z pass 2025-12-04T09:40:43.4138774Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:43.4142984Z TIMING: _recursive_pre_grad_passes:0.0076 _recursive_joint_graph_passes:0.44411 _recursive_post_grad_passes:0.06592 async_compile.wait:0.64933 code_gen:9.40248 inductor_compile:10.69216 backend_compile:14.12898 gc:0.00023 entire_frame_compile:17.09368 total_wall_time:17.09368 2025-12-04T09:40:43.4144431Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:8841 | FakeTensor.__torch_dispatch__:4457 | ProxyTorchDispatchMode.__torch_dispatch__:2621 2025-12-04T09:40:43.4144912Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-12-04T09:40:45.7981486Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:40:45.7982331Z import pynvml # type: ignore[import] 2025-12-04T09:40:49.1759109Z 2025-12-04T09:40:54.7608607Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:40:54.7608911Z loading model: 0it [00:05, ?it/s] 2025-12-04T09:40:54.7631854Z cpu eval M2M100ForConditionalGeneration 2025-12-04T09:40:55.5755805Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:55.9072500Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:40:56.2324252Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:11.6185758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6191696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6193864Z res = mod(**inputs) 2025-12-04T09:41:11.6194386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6194786Z outputs = self.model( 2025-12-04T09:41:11.6195168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6195606Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6196035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T09:41:11.6196509Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T09:41:11.6197007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:41:11.6197442Z return func(*args, **kwargs) 2025-12-04T09:41:11.6197822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:41:11.6198396Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:41:11.6199034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-12-04T09:41:11.6199498Z mask = input_ids.ne(padding_idx).int() 2025-12-04T09:41:11.6199644Z 2025-12-04T09:41:11.6199755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6200127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6200460Z res = mod(**inputs) 2025-12-04T09:41:11.6200835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6201219Z outputs = self.model( 2025-12-04T09:41:11.6201588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6202055Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6202433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T09:41:11.6202930Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T09:41:11.6203435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:41:11.6203800Z return func(*args, **kwargs) 2025-12-04T09:41:11.6204169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:41:11.6204697Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:41:11.6205278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-12-04T09:41:11.6205734Z mask = input_ids.ne(padding_idx).int() 2025-12-04T09:41:11.6205868Z 2025-12-04T09:41:11.6205950Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6206163Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6206368Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6206562Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6206761Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6206965Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6207173Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6207378Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6207592Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6207800Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6208005Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6208266Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6208539Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6208916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6209269Z res = mod(**inputs) 2025-12-04T09:41:11.6209670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6210097Z outputs = self.model( 2025-12-04T09:41:11.6210493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6210911Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6211463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T09:41:11.6212004Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T09:41:11.6212500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:41:11.6212893Z return func(*args, **kwargs) 2025-12-04T09:41:11.6213301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:41:11.6214083Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:41:11.6214668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T09:41:11.6215256Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:41:11.6215508Z 2025-12-04T09:41:11.6215639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6215995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6216322Z res = mod(**inputs) 2025-12-04T09:41:11.6216687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6217072Z outputs = self.model( 2025-12-04T09:41:11.6217430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6217815Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6218203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T09:41:11.6218647Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T09:41:11.6219072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:41:11.6219457Z return func(*args, **kwargs) 2025-12-04T09:41:11.6219863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:41:11.6220408Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:41:11.6221021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T09:41:11.6221601Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:41:11.6221849Z 2025-12-04T09:41:11.6221965Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6222330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6222667Z res = mod(**inputs) 2025-12-04T09:41:11.6223132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6223557Z outputs = self.model( 2025-12-04T09:41:11.6223935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6224341Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6224744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6225147Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6225520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6225912Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6226328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6226785Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6227214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6227626Z return func(*args, **kwargs) 2025-12-04T09:41:11.6228032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6228544Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6228765Z 2025-12-04T09:41:11.6228879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6229268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6229623Z res = mod(**inputs) 2025-12-04T09:41:11.6229993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6230386Z outputs = self.model( 2025-12-04T09:41:11.6230773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6231169Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6231550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6231945Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6232307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6232683Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6233073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6233489Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6233893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6234276Z return func(*args, **kwargs) 2025-12-04T09:41:11.6234660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6235064Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6235207Z 2025-12-04T09:41:11.6235322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6235685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6236015Z res = mod(**inputs) 2025-12-04T09:41:11.6236384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6236778Z outputs = self.model( 2025-12-04T09:41:11.6237145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6237543Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6237963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6238365Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6238709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6239060Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6239435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6239829Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6240219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6240629Z return func(*args, **kwargs) 2025-12-04T09:41:11.6241010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6241400Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6241552Z 2025-12-04T09:41:11.6241632Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6241870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6242222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6242541Z res = mod(**inputs) 2025-12-04T09:41:11.6242900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6243281Z outputs = self.model( 2025-12-04T09:41:11.6243638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6244026Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6244407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6244804Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6245172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6245550Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6245949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6246338Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6246727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6247112Z return func(*args, **kwargs) 2025-12-04T09:41:11.6247496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6247926Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6248396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6248904Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6249096Z 2025-12-04T09:41:11.6249204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6249581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6249917Z res = mod(**inputs) 2025-12-04T09:41:11.6250321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6250739Z outputs = self.model( 2025-12-04T09:41:11.6251139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6251647Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6252086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6252514Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6252886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6253283Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6253695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6254095Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6254490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6254924Z return func(*args, **kwargs) 2025-12-04T09:41:11.6255316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6255740Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6255891Z 2025-12-04T09:41:11.6256011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6256380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6256721Z res = mod(**inputs) 2025-12-04T09:41:11.6257104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6257505Z outputs = self.model( 2025-12-04T09:41:11.6257860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6258247Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6258630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6259015Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6259359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6259727Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6260108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6260534Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6260714Z 2025-12-04T09:41:11.6260817Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6261169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6261509Z res = mod(**inputs) 2025-12-04T09:41:11.6261886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6262281Z outputs = self.model( 2025-12-04T09:41:11.6262669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6263078Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6263473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6263873Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6264221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6264580Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6264976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6265436Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6265615Z 2025-12-04T09:41:11.6265734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6266140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6266490Z res = mod(**inputs) 2025-12-04T09:41:11.6266867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6267268Z outputs = self.model( 2025-12-04T09:41:11.6267640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6268060Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6268458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6268891Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6269275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6269659Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6270068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6270474Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6270625Z 2025-12-04T09:41:11.6270732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6271105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6271441Z res = mod(**inputs) 2025-12-04T09:41:11.6271813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6272214Z outputs = self.model( 2025-12-04T09:41:11.6272792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6273210Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6273611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6274026Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6274380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6274740Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6275135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6275544Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6275943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6276312Z return func(*args, **kwargs) 2025-12-04T09:41:11.6276711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6277208Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6277427Z 2025-12-04T09:41:11.6277553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6277923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6278245Z res = mod(**inputs) 2025-12-04T09:41:11.6278612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6278991Z outputs = self.model( 2025-12-04T09:41:11.6279357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6279744Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6280129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6280617Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6281003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6281373Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6281762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6282174Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6282584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6282971Z return func(*args, **kwargs) 2025-12-04T09:41:11.6283342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6283772Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6283907Z 2025-12-04T09:41:11.6284019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6284379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6284693Z res = mod(**inputs) 2025-12-04T09:41:11.6285056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6285437Z outputs = self.model( 2025-12-04T09:41:11.6285809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6286219Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6286619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6287026Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6287397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6287785Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6288207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6288631Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6289048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6289482Z return func(*args, **kwargs) 2025-12-04T09:41:11.6289876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6290292Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6290449Z 2025-12-04T09:41:11.6290537Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6290789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6291173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6291593Z res = mod(**inputs) 2025-12-04T09:41:11.6291993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6292412Z outputs = self.model( 2025-12-04T09:41:11.6292800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6293226Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6293626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6294037Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6294404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6294791Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6295268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6295705Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6296121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6296509Z return func(*args, **kwargs) 2025-12-04T09:41:11.6296903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6297328Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6297805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6298332Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6298526Z 2025-12-04T09:41:11.6298646Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6299019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6299354Z res = mod(**inputs) 2025-12-04T09:41:11.6299742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6300139Z outputs = self.model( 2025-12-04T09:41:11.6300515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6300916Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6301311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6301708Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6302075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6302455Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6302861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6303277Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6303690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6304088Z return func(*args, **kwargs) 2025-12-04T09:41:11.6304470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6304885Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6305035Z 2025-12-04T09:41:11.6305145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6305518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6305850Z res = mod(**inputs) 2025-12-04T09:41:11.6306234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6306640Z outputs = self.model( 2025-12-04T09:41:11.6307014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6307426Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6307833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6308233Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6308593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6308975Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6309411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6309881Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6310065Z 2025-12-04T09:41:11.6310175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6310549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6310890Z res = mod(**inputs) 2025-12-04T09:41:11.6311267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6311668Z outputs = self.model( 2025-12-04T09:41:11.6312051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6312474Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6312878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6313292Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6313669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6314056Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6314466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6314930Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6315108Z 2025-12-04T09:41:11.6315226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6315599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6315946Z res = mod(**inputs) 2025-12-04T09:41:11.6316336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6316747Z outputs = self.model( 2025-12-04T09:41:11.6317135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6317554Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6317958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6318363Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6318734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6319121Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6319541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6319961Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6320111Z 2025-12-04T09:41:11.6320221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6320605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6320950Z res = mod(**inputs) 2025-12-04T09:41:11.6321329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6321741Z outputs = self.model( 2025-12-04T09:41:11.6322135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6322544Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6322952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6323369Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6323761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6324131Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6324525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:41:11.6324917Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6325053Z 2025-12-04T09:41:11.6325153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6325506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6325821Z res = mod(**inputs) 2025-12-04T09:41:11.6326183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6326555Z outputs = self.model( 2025-12-04T09:41:11.6326946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6327337Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6327714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6328091Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6328437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6328814Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6329214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6329640Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6330063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6330436Z return func(*args, **kwargs) 2025-12-04T09:41:11.6330801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6331337Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6331570Z 2025-12-04T09:41:11.6331690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6332068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6332405Z res = mod(**inputs) 2025-12-04T09:41:11.6332789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6333171Z outputs = self.model( 2025-12-04T09:41:11.6333526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6333913Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6334304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6334690Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6335031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6335392Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6335778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6336172Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6336564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6336939Z return func(*args, **kwargs) 2025-12-04T09:41:11.6337309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6337729Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6337871Z 2025-12-04T09:41:11.6337992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6338350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6338672Z res = mod(**inputs) 2025-12-04T09:41:11.6339024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6339409Z outputs = self.model( 2025-12-04T09:41:11.6339776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6340156Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6340535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6340941Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6341294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6341654Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6342040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6342442Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6342824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6343195Z return func(*args, **kwargs) 2025-12-04T09:41:11.6343562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6344011Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6344153Z 2025-12-04T09:41:11.6344232Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6344470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6344826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6345138Z res = mod(**inputs) 2025-12-04T09:41:11.6345494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6345873Z outputs = self.model( 2025-12-04T09:41:11.6346234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6346612Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6346988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6347370Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6347718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6348073Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6348457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6348857Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6349239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6349613Z return func(*args, **kwargs) 2025-12-04T09:41:11.6349981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6350444Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6350867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6351332Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6351542Z 2025-12-04T09:41:11.6351669Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6352019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6352321Z res = mod(**inputs) 2025-12-04T09:41:11.6352672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6353045Z outputs = self.model( 2025-12-04T09:41:11.6353389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6353762Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6354128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6354524Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6354861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6355217Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6355599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6355985Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6356370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6356737Z return func(*args, **kwargs) 2025-12-04T09:41:11.6357101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6357478Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6357621Z 2025-12-04T09:41:11.6357721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6358071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6358385Z res = mod(**inputs) 2025-12-04T09:41:11.6358729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6359105Z outputs = self.model( 2025-12-04T09:41:11.6359459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6359829Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6360201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6360574Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6360913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6361258Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6361635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6362057Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6362222Z 2025-12-04T09:41:11.6362327Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6362666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6362977Z res = mod(**inputs) 2025-12-04T09:41:11.6363328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6363695Z outputs = self.model( 2025-12-04T09:41:11.6364051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6364427Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6364828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6365216Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6365549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6365892Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6366248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6366649Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6366812Z 2025-12-04T09:41:11.6366909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6367253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6367575Z res = mod(**inputs) 2025-12-04T09:41:11.6367926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6368298Z outputs = self.model( 2025-12-04T09:41:11.6368647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6369013Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6369377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6369747Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6370075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6370425Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6370796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6371178Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6371374Z 2025-12-04T09:41:11.6371484Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6371839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6372166Z res = mod(**inputs) 2025-12-04T09:41:11.6372725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6373151Z outputs = self.model( 2025-12-04T09:41:11.6373543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6373942Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6374316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6374690Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6375026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6375376Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6375743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6376134Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6376513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6376864Z return func(*args, **kwargs) 2025-12-04T09:41:11.6377217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6377656Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6377852Z 2025-12-04T09:41:11.6377955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6378361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6378705Z res = mod(**inputs) 2025-12-04T09:41:11.6379049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6379411Z outputs = self.model( 2025-12-04T09:41:11.6379749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6380115Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6380498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6380858Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6381218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6381563Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6381932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6382307Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6382679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6383032Z return func(*args, **kwargs) 2025-12-04T09:41:11.6383380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6383750Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6383883Z 2025-12-04T09:41:11.6383979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6384316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6384609Z res = mod(**inputs) 2025-12-04T09:41:11.6384953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6385314Z outputs = self.model( 2025-12-04T09:41:11.6385654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6386013Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6386369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6386734Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6387056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6387399Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6387766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6388148Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6388513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6388866Z return func(*args, **kwargs) 2025-12-04T09:41:11.6389219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6389589Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6389738Z 2025-12-04T09:41:11.6389812Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6390032Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6390362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6390651Z res = mod(**inputs) 2025-12-04T09:41:11.6391014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6391383Z outputs = self.model( 2025-12-04T09:41:11.6391707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6392073Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6392420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6392773Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6393084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6393416Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6393771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6394159Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6394517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6394864Z return func(*args, **kwargs) 2025-12-04T09:41:11.6395208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6395577Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6395984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6396427Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6396594Z 2025-12-04T09:41:11.6396695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6397019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6397314Z res = mod(**inputs) 2025-12-04T09:41:11.6397652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6398009Z outputs = self.model( 2025-12-04T09:41:11.6398344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6398708Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6399066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6399421Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6399748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6400091Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6400452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6400828Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6401199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6401553Z return func(*args, **kwargs) 2025-12-04T09:41:11.6401898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6402269Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6402404Z 2025-12-04T09:41:11.6402502Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6402839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6403132Z res = mod(**inputs) 2025-12-04T09:41:11.6403473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6403835Z outputs = self.model( 2025-12-04T09:41:11.6404215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6404599Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6404964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6405338Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6405667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6406017Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6406389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6406822Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6406987Z 2025-12-04T09:41:11.6407088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6407438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6407752Z res = mod(**inputs) 2025-12-04T09:41:11.6408106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6408471Z outputs = self.model( 2025-12-04T09:41:11.6408826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6409204Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6409563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6409940Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6410279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6410632Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6411016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6411524Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6411698Z 2025-12-04T09:41:11.6411818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6412202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6412552Z res = mod(**inputs) 2025-12-04T09:41:11.6412906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6413285Z outputs = self.model( 2025-12-04T09:41:11.6413640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6414034Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6414400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6414767Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6415091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6415433Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6415800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6416169Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6416306Z 2025-12-04T09:41:11.6416401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6416737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6417039Z res = mod(**inputs) 2025-12-04T09:41:11.6417414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6417797Z outputs = self.model( 2025-12-04T09:41:11.6418140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6418503Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6418861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6419220Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6419547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6419887Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6420278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:41:11.6420659Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6420789Z 2025-12-04T09:41:11.6420896Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6421232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6421539Z res = mod(**inputs) 2025-12-04T09:41:11.6421883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6422240Z outputs = self.model( 2025-12-04T09:41:11.6422584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6422954Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6423319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6423684Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6424018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6424369Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6424807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6425198Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6425576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6425933Z return func(*args, **kwargs) 2025-12-04T09:41:11.6426283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6426726Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6426926Z 2025-12-04T09:41:11.6427024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6427363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6427670Z res = mod(**inputs) 2025-12-04T09:41:11.6428004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6428359Z outputs = self.model( 2025-12-04T09:41:11.6428687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6429048Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6429400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6429758Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6430126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6430475Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6430862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6431248Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6431602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6431947Z return func(*args, **kwargs) 2025-12-04T09:41:11.6432296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6432661Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6432802Z 2025-12-04T09:41:11.6432916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6433253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6433557Z res = mod(**inputs) 2025-12-04T09:41:11.6433896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6434271Z outputs = self.model( 2025-12-04T09:41:11.6434625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6435001Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6435371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6435744Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6436080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6436426Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6436814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6437200Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6437580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6437937Z return func(*args, **kwargs) 2025-12-04T09:41:11.6438304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6438689Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6438826Z 2025-12-04T09:41:11.6438906Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6439142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6439490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6439806Z res = mod(**inputs) 2025-12-04T09:41:11.6440149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6440522Z outputs = self.model( 2025-12-04T09:41:11.6440876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6441249Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6441620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6441993Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6442330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6442679Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6443062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6443581Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6443974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6444326Z return func(*args, **kwargs) 2025-12-04T09:41:11.6444682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6445119Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6445534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6445995Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6446179Z 2025-12-04T09:41:11.6446279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6446644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6446959Z res = mod(**inputs) 2025-12-04T09:41:11.6447305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6447676Z outputs = self.model( 2025-12-04T09:41:11.6448020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6448400Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6448770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6449144Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6449476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6449833Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6450210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6450604Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6450979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6451444Z return func(*args, **kwargs) 2025-12-04T09:41:11.6451855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6452289Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6452446Z 2025-12-04T09:41:11.6452558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6452953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6453266Z res = mod(**inputs) 2025-12-04T09:41:11.6453607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6453973Z outputs = self.model( 2025-12-04T09:41:11.6454327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6454696Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6455049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6455413Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6455743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6456082Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6456452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6456866Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6457028Z 2025-12-04T09:41:11.6457166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6457519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6457829Z res = mod(**inputs) 2025-12-04T09:41:11.6458173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6458529Z outputs = self.model( 2025-12-04T09:41:11.6458890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6459259Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6459617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6460041Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6460375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6460782Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6461159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6461565Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6461736Z 2025-12-04T09:41:11.6461832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6462173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6462475Z res = mod(**inputs) 2025-12-04T09:41:11.6462827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6463258Z outputs = self.model( 2025-12-04T09:41:11.6463611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6463981Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6464345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6464719Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6465048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6465394Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6465770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6466146Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6466274Z 2025-12-04T09:41:11.6466368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6466713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6467025Z res = mod(**inputs) 2025-12-04T09:41:11.6467371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6467731Z outputs = self.model( 2025-12-04T09:41:11.6468082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6468460Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6468815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6469186Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6469521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6469869Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6470264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6470662Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6471033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6471380Z return func(*args, **kwargs) 2025-12-04T09:41:11.6471721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6472145Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6472491Z 2025-12-04T09:41:11.6472605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6472945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6473301Z res = mod(**inputs) 2025-12-04T09:41:11.6473651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6474016Z outputs = self.model( 2025-12-04T09:41:11.6474353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6474729Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6475076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6475424Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6475749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6476089Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6476448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6476819Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6477200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6477554Z return func(*args, **kwargs) 2025-12-04T09:41:11.6477905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6478270Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6478402Z 2025-12-04T09:41:11.6478501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6478841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6479141Z res = mod(**inputs) 2025-12-04T09:41:11.6479482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6479853Z outputs = self.model( 2025-12-04T09:41:11.6480195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6480546Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6480898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6481255Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6481570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6481905Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6482262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6482635Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6482992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6483402Z return func(*args, **kwargs) 2025-12-04T09:41:11.6483768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6484133Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6484263Z 2025-12-04T09:41:11.6484338Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6484566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6484904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6485200Z res = mod(**inputs) 2025-12-04T09:41:11.6485548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6485917Z outputs = self.model( 2025-12-04T09:41:11.6486291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6486665Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6487042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6487424Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6487768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6488132Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6488521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6488921Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6489308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6489676Z return func(*args, **kwargs) 2025-12-04T09:41:11.6490052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6490460Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6490898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6491440Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6491638Z 2025-12-04T09:41:11.6491754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6492127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6492494Z res = mod(**inputs) 2025-12-04T09:41:11.6492881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6493254Z outputs = self.model( 2025-12-04T09:41:11.6493603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6493979Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6494351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6494726Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6495058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6497944Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6498335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6498735Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6499123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6499513Z return func(*args, **kwargs) 2025-12-04T09:41:11.6499879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6500251Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6500381Z 2025-12-04T09:41:11.6500487Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6500825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6501121Z res = mod(**inputs) 2025-12-04T09:41:11.6501490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6501848Z outputs = self.model( 2025-12-04T09:41:11.6502183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6502582Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6502934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6503278Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6503599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6503933Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6504287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6504678Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6504842Z 2025-12-04T09:41:11.6504939Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6505268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6505565Z res = mod(**inputs) 2025-12-04T09:41:11.6505893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6506250Z outputs = self.model( 2025-12-04T09:41:11.6506585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6506942Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6507291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6507646Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6507968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6508299Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6508658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6509055Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6509213Z 2025-12-04T09:41:11.6509312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6509633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6509929Z res = mod(**inputs) 2025-12-04T09:41:11.6510258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6510603Z outputs = self.model( 2025-12-04T09:41:11.6511011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6511372Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6511723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6512068Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6512404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6512753Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6513102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6513464Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6513595Z 2025-12-04T09:41:11.6513688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6514019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6514306Z res = mod(**inputs) 2025-12-04T09:41:11.6514636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6515006Z outputs = self.model( 2025-12-04T09:41:11.6515332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6515690Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6516040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6516393Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6516707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6517042Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6517397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:41:11.6517751Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6517875Z 2025-12-04T09:41:11.6517967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6518296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6518589Z res = mod(**inputs) 2025-12-04T09:41:11.6518911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6519262Z outputs = self.model( 2025-12-04T09:41:11.6519593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6519944Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6520286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6520639Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6520958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6521286Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6521643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6522015Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6522380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6522716Z return func(*args, **kwargs) 2025-12-04T09:41:11.6523063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6523517Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6523707Z 2025-12-04T09:41:11.6523810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6524136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6524434Z res = mod(**inputs) 2025-12-04T09:41:11.6524778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6525143Z outputs = self.model( 2025-12-04T09:41:11.6525480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6525838Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6526187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6526548Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6526874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6527209Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6527595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6527961Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6528327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6528669Z return func(*args, **kwargs) 2025-12-04T09:41:11.6529005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6529366Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6529496Z 2025-12-04T09:41:11.6529591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6529919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6530206Z res = mod(**inputs) 2025-12-04T09:41:11.6530541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6530897Z outputs = self.model( 2025-12-04T09:41:11.6531226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6531693Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6532099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6532513Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6532881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6533250Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6533646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6534039Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6534409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6534774Z return func(*args, **kwargs) 2025-12-04T09:41:11.6535171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6535589Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6535745Z 2025-12-04T09:41:11.6535832Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6536085Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6536488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6536829Z res = mod(**inputs) 2025-12-04T09:41:11.6537221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6537650Z outputs = self.model( 2025-12-04T09:41:11.6538058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6538489Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6538870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6539222Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6539537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6539878Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6540237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6540607Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6540984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6541329Z return func(*args, **kwargs) 2025-12-04T09:41:11.6541671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6542044Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6542457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6542895Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6543063Z 2025-12-04T09:41:11.6543166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6543489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6543788Z res = mod(**inputs) 2025-12-04T09:41:11.6544122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6544473Z outputs = self.model( 2025-12-04T09:41:11.6544802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6545158Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6545507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6545854Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6546175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6546513Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6546870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6547231Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6547590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6547933Z return func(*args, **kwargs) 2025-12-04T09:41:11.6548265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6548626Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6548757Z 2025-12-04T09:41:11.6548850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6549180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6549486Z res = mod(**inputs) 2025-12-04T09:41:11.6549820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6550169Z outputs = self.model( 2025-12-04T09:41:11.6550502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6550864Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6551230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6551582Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6551900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6552239Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6552599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6552996Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6553151Z 2025-12-04T09:41:11.6553245Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6553598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6553904Z res = mod(**inputs) 2025-12-04T09:41:11.6554235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6554596Z outputs = self.model( 2025-12-04T09:41:11.6554938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6555298Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6555651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6556015Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6556348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6556696Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6557055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6557459Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6557618Z 2025-12-04T09:41:11.6557725Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6558063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6558420Z res = mod(**inputs) 2025-12-04T09:41:11.6558759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6559122Z outputs = self.model( 2025-12-04T09:41:11.6559460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6559825Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6560182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6560540Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6560861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6561203Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6561567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6561930Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6562066Z 2025-12-04T09:41:11.6562184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6562521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6562822Z res = mod(**inputs) 2025-12-04T09:41:11.6563156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6563517Z outputs = self.model( 2025-12-04T09:41:11.6563872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6564244Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6564593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6564948Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6565270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6565598Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6565956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6566345Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6566707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6567049Z return func(*args, **kwargs) 2025-12-04T09:41:11.6567402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6567838Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6568032Z 2025-12-04T09:41:11.6568129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6568468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6568772Z res = mod(**inputs) 2025-12-04T09:41:11.6569116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6569479Z outputs = self.model( 2025-12-04T09:41:11.6569826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6570214Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6570569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6570644Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6570855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6570937Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6571181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6571327Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6571575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6571647Z return func(*args, **kwargs) 2025-12-04T09:41:11.6571895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6571974Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6571978Z 2025-12-04T09:41:11.6572078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6572439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6572510Z res = mod(**inputs) 2025-12-04T09:41:11.6572816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6572891Z outputs = self.model( 2025-12-04T09:41:11.6573138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6573220Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6573486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6573591Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6573816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6573890Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6574140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6574229Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6574472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6574544Z return func(*args, **kwargs) 2025-12-04T09:41:11.6574806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6574886Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6574890Z 2025-12-04T09:41:11.6574974Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6575070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6575263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6575323Z res = mod(**inputs) 2025-12-04T09:41:11.6575567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6575641Z outputs = self.model( 2025-12-04T09:41:11.6575883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6575951Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6576198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6576266Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6576483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6576556Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6576791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6576882Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6577111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6577182Z return func(*args, **kwargs) 2025-12-04T09:41:11.6577417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6577510Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6577792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6577916Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6577919Z 2025-12-04T09:41:11.6578014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6578206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6578266Z res = mod(**inputs) 2025-12-04T09:41:11.6578526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6578593Z outputs = self.model( 2025-12-04T09:41:11.6578832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6578909Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6579163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6579251Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6579460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6579534Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6579778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6579862Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6580093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6580166Z return func(*args, **kwargs) 2025-12-04T09:41:11.6580401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6580501Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6580506Z 2025-12-04T09:41:11.6580602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6580784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6580849Z res = mod(**inputs) 2025-12-04T09:41:11.6581088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6581156Z outputs = self.model( 2025-12-04T09:41:11.6581395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6581461Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6581701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6581769Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6581977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6582056Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6582290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6582405Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6582409Z 2025-12-04T09:41:11.6582504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6582690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6582755Z res = mod(**inputs) 2025-12-04T09:41:11.6582998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6583062Z outputs = self.model( 2025-12-04T09:41:11.6583308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6583375Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6583617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6583682Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6583891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6583970Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6584226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6584347Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6584352Z 2025-12-04T09:41:11.6584450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6584653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6584755Z res = mod(**inputs) 2025-12-04T09:41:11.6585004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6585068Z outputs = self.model( 2025-12-04T09:41:11.6585320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6585387Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6585636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6585701Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6585910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6586006Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6586247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6586333Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6586336Z 2025-12-04T09:41:11.6586432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6586615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6586683Z res = mod(**inputs) 2025-12-04T09:41:11.6586926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6586989Z outputs = self.model( 2025-12-04T09:41:11.6587237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6587306Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6587554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6587621Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6587830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6587908Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6588146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:41:11.6588227Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6588232Z 2025-12-04T09:41:11.6588328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6588512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6588581Z res = mod(**inputs) 2025-12-04T09:41:11.6588819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6588896Z outputs = self.model( 2025-12-04T09:41:11.6589144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6589211Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6589453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6589518Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6589738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6589822Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6590063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6590149Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6590406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6590492Z return func(*args, **kwargs) 2025-12-04T09:41:11.6590737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6590877Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6590881Z 2025-12-04T09:41:11.6591552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6591747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6591807Z res = mod(**inputs) 2025-12-04T09:41:11.6592055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6592134Z outputs = self.model( 2025-12-04T09:41:11.6592376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6592454Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6592689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6592763Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6592971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6593044Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6593286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6593371Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6593602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6593678Z return func(*args, **kwargs) 2025-12-04T09:41:11.6593914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6593997Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6594001Z 2025-12-04T09:41:11.6594098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6594283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6594348Z res = mod(**inputs) 2025-12-04T09:41:11.6594587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6594649Z outputs = self.model( 2025-12-04T09:41:11.6594900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6594968Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6595202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6595266Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6595467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6595544Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6595770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6595874Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6596101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6596166Z return func(*args, **kwargs) 2025-12-04T09:41:11.6596403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6596505Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6596522Z 2025-12-04T09:41:11.6596598Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6596699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6596877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6596941Z res = mod(**inputs) 2025-12-04T09:41:11.6597173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6597235Z outputs = self.model( 2025-12-04T09:41:11.6597476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6597557Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6597798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6597866Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6598069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6598149Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6598381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6598463Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6598698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6598761Z return func(*args, **kwargs) 2025-12-04T09:41:11.6598997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6599087Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6599354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6599484Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6599487Z 2025-12-04T09:41:11.6599579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6599767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6599824Z res = mod(**inputs) 2025-12-04T09:41:11.6600060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6600129Z outputs = self.model( 2025-12-04T09:41:11.6600362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6600430Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6600670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6600737Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6600947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6601018Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6601247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6602090Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6602318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6602480Z return func(*args, **kwargs) 2025-12-04T09:41:11.6602763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6602865Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6602884Z 2025-12-04T09:41:11.6602991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6603172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6603229Z res = mod(**inputs) 2025-12-04T09:41:11.6603470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6603533Z outputs = self.model( 2025-12-04T09:41:11.6603777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6603845Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6604097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6604172Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6604374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6604445Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6604680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6604786Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6604789Z 2025-12-04T09:41:11.6604888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6605067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6605123Z res = mod(**inputs) 2025-12-04T09:41:11.6605359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6605421Z outputs = self.model( 2025-12-04T09:41:11.6605659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6605725Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6605952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6606024Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6606224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6606293Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6606529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6606633Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6606638Z 2025-12-04T09:41:11.6606740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6606917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6606976Z res = mod(**inputs) 2025-12-04T09:41:11.6607215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6607276Z outputs = self.model( 2025-12-04T09:41:11.6607513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6607596Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6607833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6607906Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6608111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6608194Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6608436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6608526Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6608530Z 2025-12-04T09:41:11.6608632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6608811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6608869Z res = mod(**inputs) 2025-12-04T09:41:11.6609107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6609167Z outputs = self.model( 2025-12-04T09:41:11.6609398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6609487Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6609725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6609796Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6610001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6610070Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6610310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6610392Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6610625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6610688Z return func(*args, **kwargs) 2025-12-04T09:41:11.6610920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6611065Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6611070Z 2025-12-04T09:41:11.6611163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6611394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6611462Z res = mod(**inputs) 2025-12-04T09:41:11.6611725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6611804Z outputs = self.model( 2025-12-04T09:41:11.6612070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6612143Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6612415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6612489Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6612730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6612811Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6613077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6613172Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6613427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6613496Z return func(*args, **kwargs) 2025-12-04T09:41:11.6613757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6613837Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6613840Z 2025-12-04T09:41:11.6613963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6614171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6614238Z res = mod(**inputs) 2025-12-04T09:41:11.6614520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6614590Z outputs = self.model( 2025-12-04T09:41:11.6614867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6614942Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6615211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6615311Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6615554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6615634Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6615913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6616005Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6616277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6616348Z return func(*args, **kwargs) 2025-12-04T09:41:11.6616622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6616719Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6616723Z 2025-12-04T09:41:11.6616809Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6616915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6617134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6617202Z res = mod(**inputs) 2025-12-04T09:41:11.6617485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6617553Z outputs = self.model( 2025-12-04T09:41:11.6617836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6617918Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6618190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6618271Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6618510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6618592Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6618870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6618965Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6619233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6619310Z return func(*args, **kwargs) 2025-12-04T09:41:11.6619584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6619719Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6620034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6620175Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6620179Z 2025-12-04T09:41:11.6620319Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6620518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6620585Z res = mod(**inputs) 2025-12-04T09:41:11.6620827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6620888Z outputs = self.model( 2025-12-04T09:41:11.6621135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6621203Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6621439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6621534Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6621747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6621826Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6622070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6622152Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6622387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6622454Z return func(*args, **kwargs) 2025-12-04T09:41:11.6622692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6622774Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6622778Z 2025-12-04T09:41:11.6622872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6623065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6623125Z res = mod(**inputs) 2025-12-04T09:41:11.6623372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6623440Z outputs = self.model( 2025-12-04T09:41:11.6623682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6623755Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6623993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6624058Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6624274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6624346Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6624586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6624705Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6624709Z 2025-12-04T09:41:11.6624803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6624993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6625051Z res = mod(**inputs) 2025-12-04T09:41:11.6625309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6625381Z outputs = self.model( 2025-12-04T09:41:11.6625625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6625701Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6625958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6626040Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6626255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6626327Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6626563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6626682Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6626686Z 2025-12-04T09:41:11.6626784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6626974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6627034Z res = mod(**inputs) 2025-12-04T09:41:11.6627288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6627360Z outputs = self.model( 2025-12-04T09:41:11.6627598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6627670Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6627905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6627971Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6628184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6628255Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6628490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6628576Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6628579Z 2025-12-04T09:41:11.6628675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6628871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6628931Z res = mod(**inputs) 2025-12-04T09:41:11.6629173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6629245Z outputs = self.model( 2025-12-04T09:41:11.6629538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6629606Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6629851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6629918Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6630133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6630204Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6630442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:41:11.6630523Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6630526Z 2025-12-04T09:41:11.6630620Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6630811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6630886Z res = mod(**inputs) 2025-12-04T09:41:11.6631191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6631261Z outputs = self.model( 2025-12-04T09:41:11.6631503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6631586Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6631861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6631928Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6632144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6632215Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6632449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6632539Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6632768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6632859Z return func(*args, **kwargs) 2025-12-04T09:41:11.6633100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6633243Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6633247Z 2025-12-04T09:41:11.6633350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6633534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6633592Z res = mod(**inputs) 2025-12-04T09:41:11.6633844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6633905Z outputs = self.model( 2025-12-04T09:41:11.6634154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6634223Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6634460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6634537Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6634746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6634823Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6635062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6635144Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6635383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6635448Z return func(*args, **kwargs) 2025-12-04T09:41:11.6635685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6635769Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6635773Z 2025-12-04T09:41:11.6635869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6636063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6636123Z res = mod(**inputs) 2025-12-04T09:41:11.6636364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6636435Z outputs = self.model( 2025-12-04T09:41:11.6636717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6636784Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6637029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6637097Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6637328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6637415Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6637648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6637738Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6637968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6638039Z return func(*args, **kwargs) 2025-12-04T09:41:11.6638274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6638353Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6638380Z 2025-12-04T09:41:11.6638463Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6638560Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6638745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6638813Z res = mod(**inputs) 2025-12-04T09:41:11.6639053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6639122Z outputs = self.model( 2025-12-04T09:41:11.6639363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6639432Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6639678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6639745Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6639964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6640037Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6640278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6640369Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6640597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6640661Z return func(*args, **kwargs) 2025-12-04T09:41:11.6640906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6640996Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6641274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6641402Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6641407Z 2025-12-04T09:41:11.6641501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6641695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6641755Z res = mod(**inputs) 2025-12-04T09:41:11.6642003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6642066Z outputs = self.model( 2025-12-04T09:41:11.6642322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6642400Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6642636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6642705Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6642936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6643025Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6643269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6643351Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6643581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6643657Z return func(*args, **kwargs) 2025-12-04T09:41:11.6643897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6643971Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6643997Z 2025-12-04T09:41:11.6644096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6644282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6644351Z res = mod(**inputs) 2025-12-04T09:41:11.6644592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6644654Z outputs = self.model( 2025-12-04T09:41:11.6644905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6644975Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6645231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6645301Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6645517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6645601Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6645845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6645971Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6645982Z 2025-12-04T09:41:11.6646082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6646279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6646346Z res = mod(**inputs) 2025-12-04T09:41:11.6646597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6646663Z outputs = self.model( 2025-12-04T09:41:11.6646924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6646999Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6647259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6647330Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6647551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6647634Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6647885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6648013Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6648024Z 2025-12-04T09:41:11.6648126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6648320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6648391Z res = mod(**inputs) 2025-12-04T09:41:11.6648678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6648764Z outputs = self.model( 2025-12-04T09:41:11.6649046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6649122Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6649398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6649471Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6649712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6649801Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6650071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6650173Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6650178Z 2025-12-04T09:41:11.6650294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6650500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6650571Z res = mod(**inputs) 2025-12-04T09:41:11.6650837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6650906Z outputs = self.model( 2025-12-04T09:41:11.6651183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6651321Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6651600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6651686Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6651920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6652011Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6652277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6652373Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6652637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6652713Z return func(*args, **kwargs) 2025-12-04T09:41:11.6652987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6653135Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6653141Z 2025-12-04T09:41:11.6653243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6653446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6653510Z res = mod(**inputs) 2025-12-04T09:41:11.6653763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6653838Z outputs = self.model( 2025-12-04T09:41:11.6654092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6654171Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6654448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6654518Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6654744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6654832Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6655083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6655188Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6655419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6655493Z return func(*args, **kwargs) 2025-12-04T09:41:11.6655737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6655813Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6655824Z 2025-12-04T09:41:11.6655923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6656134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6656203Z res = mod(**inputs) 2025-12-04T09:41:11.6656449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6656517Z outputs = self.model( 2025-12-04T09:41:11.6656771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6656840Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6657088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6657158Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6657372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6657453Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6657697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6657782Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6658025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6658092Z return func(*args, **kwargs) 2025-12-04T09:41:11.6658338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6658420Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6658423Z 2025-12-04T09:41:11.6658502Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6658607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6658795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6658857Z res = mod(**inputs) 2025-12-04T09:41:11.6659109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6659173Z outputs = self.model( 2025-12-04T09:41:11.6659425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6659493Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6659733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6659809Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6660041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6660124Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6660369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6660457Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6660721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6660813Z return func(*args, **kwargs) 2025-12-04T09:41:11.6661052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6661154Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6661434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6661570Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6661574Z 2025-12-04T09:41:11.6661672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6661859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6661944Z res = mod(**inputs) 2025-12-04T09:41:11.6662202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6662275Z outputs = self.model( 2025-12-04T09:41:11.6662528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6662599Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6662856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6662926Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6663143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6663224Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6663478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T09:41:11.6663575Z hidden_states, attn_weights = self.self_attn( 2025-12-04T09:41:11.6663820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6663888Z return func(*args, **kwargs) 2025-12-04T09:41:11.6664144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6664221Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6664224Z 2025-12-04T09:41:11.6664332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6664526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6664588Z res = mod(**inputs) 2025-12-04T09:41:11.6664845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6664912Z outputs = self.model( 2025-12-04T09:41:11.6665163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6665242Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6665491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6665567Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6665787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6665878Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6666127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6666241Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6666245Z 2025-12-04T09:41:11.6666350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6666551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6666629Z res = mod(**inputs) 2025-12-04T09:41:11.6666880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6666946Z outputs = self.model( 2025-12-04T09:41:11.6667195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6667274Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6667515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6667589Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6667821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6667895Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6668147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T09:41:11.6668261Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6668264Z 2025-12-04T09:41:11.6668363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6668559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6668618Z res = mod(**inputs) 2025-12-04T09:41:11.6668874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6668937Z outputs = self.model( 2025-12-04T09:41:11.6669187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6669264Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6669507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6669593Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6669804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6669877Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6670120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T09:41:11.6670193Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6670197Z 2025-12-04T09:41:11.6670291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6670487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6670548Z res = mod(**inputs) 2025-12-04T09:41:11.6670797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6670861Z outputs = self.model( 2025-12-04T09:41:11.6671100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T09:41:11.6671173Z encoder_outputs = self.encoder( 2025-12-04T09:41:11.6671412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T09:41:11.6671501Z layer_outputs = encoder_layer( 2025-12-04T09:41:11.6671713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6671788Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6672033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T09:41:11.6672125Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6672143Z 2025-12-04T09:41:11.6672242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6672549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6672613Z res = mod(**inputs) 2025-12-04T09:41:11.6672864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6672930Z outputs = self.model( 2025-12-04T09:41:11.6673175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6673254Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6673499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T09:41:11.6673699Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T09:41:11.6673935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:41:11.6674003Z return func(*args, **kwargs) 2025-12-04T09:41:11.6674248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:41:11.6674450Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:41:11.6674754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T09:41:11.6674940Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:41:11.6674946Z 2025-12-04T09:41:11.6675043Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6675238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6675301Z res = mod(**inputs) 2025-12-04T09:41:11.6675552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6675625Z outputs = self.model( 2025-12-04T09:41:11.6675866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6675943Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6676184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T09:41:11.6676341Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T09:41:11.6676575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T09:41:11.6676643Z return func(*args, **kwargs) 2025-12-04T09:41:11.6676882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T09:41:11.6677090Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T09:41:11.6677385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T09:41:11.6677594Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:41:11.6677598Z 2025-12-04T09:41:11.6677696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6677881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6677949Z res = mod(**inputs) 2025-12-04T09:41:11.6678211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6678302Z outputs = self.model( 2025-12-04T09:41:11.6678547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6678616Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6678863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6678933Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6679154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6679229Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6679478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6679553Z return func(*args, **kwargs) 2025-12-04T09:41:11.6679793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6679888Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6680125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6680189Z return func(*args, **kwargs) 2025-12-04T09:41:11.6680435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6680576Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6680579Z 2025-12-04T09:41:11.6680674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6680872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6680931Z res = mod(**inputs) 2025-12-04T09:41:11.6681175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6681238Z outputs = self.model( 2025-12-04T09:41:11.6681477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6681551Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6681788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6681856Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6682082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6682155Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6682383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6682446Z return func(*args, **kwargs) 2025-12-04T09:41:11.6682678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6682776Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6683000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6683063Z return func(*args, **kwargs) 2025-12-04T09:41:11.6683316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6683390Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6683394Z 2025-12-04T09:41:11.6683494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6683673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6683754Z res = mod(**inputs) 2025-12-04T09:41:11.6684012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6684072Z outputs = self.model( 2025-12-04T09:41:11.6684310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6684376Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6684607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6684680Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6684885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6684972Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6685203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6685266Z return func(*args, **kwargs) 2025-12-04T09:41:11.6685512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6685603Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6685826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6685910Z return func(*args, **kwargs) 2025-12-04T09:41:11.6686138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6686223Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6686228Z 2025-12-04T09:41:11.6686303Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6686394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6686581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6686639Z res = mod(**inputs) 2025-12-04T09:41:11.6686872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6686942Z outputs = self.model( 2025-12-04T09:41:11.6687173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6687247Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6687477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6687544Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6687760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6687831Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6688051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6688120Z return func(*args, **kwargs) 2025-12-04T09:41:11.6688349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6688442Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6688677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6688742Z return func(*args, **kwargs) 2025-12-04T09:41:11.6688980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6689071Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6689356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6689491Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6689495Z 2025-12-04T09:41:11.6689588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6689774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6689832Z res = mod(**inputs) 2025-12-04T09:41:11.6690069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6690138Z outputs = self.model( 2025-12-04T09:41:11.6690368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6690458Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6690695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6690762Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6690974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6691044Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6691342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6691424Z return func(*args, **kwargs) 2025-12-04T09:41:11.6691693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6691804Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6692059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6692131Z return func(*args, **kwargs) 2025-12-04T09:41:11.6692408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6692504Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6692508Z 2025-12-04T09:41:11.6692614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6692805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6692865Z res = mod(**inputs) 2025-12-04T09:41:11.6693120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6693184Z outputs = self.model( 2025-12-04T09:41:11.6693430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6693510Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6693756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6693835Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6694059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6694130Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6694368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6694459Z return func(*args, **kwargs) 2025-12-04T09:41:11.6694697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6694796Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6695018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6695102Z return func(*args, **kwargs) 2025-12-04T09:41:11.6695359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6695498Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6695509Z 2025-12-04T09:41:11.6695604Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6695791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6695860Z res = mod(**inputs) 2025-12-04T09:41:11.6696099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6696162Z outputs = self.model( 2025-12-04T09:41:11.6696427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6696494Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6696733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6696798Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6697000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6697076Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6697298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6697359Z return func(*args, **kwargs) 2025-12-04T09:41:11.6697594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6697691Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6697922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6697986Z return func(*args, **kwargs) 2025-12-04T09:41:11.6698216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6698294Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6698297Z 2025-12-04T09:41:11.6698390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6698581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6698639Z res = mod(**inputs) 2025-12-04T09:41:11.6698870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6698939Z outputs = self.model( 2025-12-04T09:41:11.6699170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6699237Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6699477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6699541Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6699751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6699820Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6700058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6700131Z return func(*args, **kwargs) 2025-12-04T09:41:11.6700364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6700463Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6700710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6700788Z return func(*args, **kwargs) 2025-12-04T09:41:11.6701027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6701108Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6701111Z 2025-12-04T09:41:11.6701185Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6701287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6701469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6701536Z res = mod(**inputs) 2025-12-04T09:41:11.6701768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6701846Z outputs = self.model( 2025-12-04T09:41:11.6702093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6702164Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6702400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6702474Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6702678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6702759Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6702983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6703045Z return func(*args, **kwargs) 2025-12-04T09:41:11.6703287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6703385Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6703621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6703684Z return func(*args, **kwargs) 2025-12-04T09:41:11.6703915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6704012Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6704278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6704400Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6704411Z 2025-12-04T09:41:11.6704506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6704690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6704754Z res = mod(**inputs) 2025-12-04T09:41:11.6704990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6705051Z outputs = self.model( 2025-12-04T09:41:11.6705293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6705359Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6705625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6705693Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6705896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6705976Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6706211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6706288Z return func(*args, **kwargs) 2025-12-04T09:41:11.6706546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6706642Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6706880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6706945Z return func(*args, **kwargs) 2025-12-04T09:41:11.6707180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6707261Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6707278Z 2025-12-04T09:41:11.6707373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6707562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6707623Z res = mod(**inputs) 2025-12-04T09:41:11.6707857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6707926Z outputs = self.model( 2025-12-04T09:41:11.6708159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6708225Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6708465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6708531Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6708742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6708815Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6709039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6709112Z return func(*args, **kwargs) 2025-12-04T09:41:11.6709342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6709451Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6709460Z 2025-12-04T09:41:11.6709552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6709735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6709799Z res = mod(**inputs) 2025-12-04T09:41:11.6710035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6710097Z outputs = self.model( 2025-12-04T09:41:11.6710338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6710407Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6710648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6710713Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6710916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6711008Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6711233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6711297Z return func(*args, **kwargs) 2025-12-04T09:41:11.6711536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6711658Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6711675Z 2025-12-04T09:41:11.6711779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6711962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6712020Z res = mod(**inputs) 2025-12-04T09:41:11.6712264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6712325Z outputs = self.model( 2025-12-04T09:41:11.6712569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6712635Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6712871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6712960Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6713168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6713241Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6713475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6713537Z return func(*args, **kwargs) 2025-12-04T09:41:11.6713779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.6713854Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6713857Z 2025-12-04T09:41:11.6713951Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6714141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6714202Z res = mod(**inputs) 2025-12-04T09:41:11.6714447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6714517Z outputs = self.model( 2025-12-04T09:41:11.6714762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6714836Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6715078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6715145Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6715363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6715435Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6715676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6715739Z return func(*args, **kwargs) 2025-12-04T09:41:11.6715984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6716085Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6716316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6716380Z return func(*args, **kwargs) 2025-12-04T09:41:11.6716643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6716788Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6716792Z 2025-12-04T09:41:11.6716898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6717086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6717145Z res = mod(**inputs) 2025-12-04T09:41:11.6717412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6717489Z outputs = self.model( 2025-12-04T09:41:11.6717737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6717804Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6718046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6718119Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6718328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6718420Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6718659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6718722Z return func(*args, **kwargs) 2025-12-04T09:41:11.6718968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6719061Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6719289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6719361Z return func(*args, **kwargs) 2025-12-04T09:41:11.6719600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6719681Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6719684Z 2025-12-04T09:41:11.6719782Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6719967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6720034Z res = mod(**inputs) 2025-12-04T09:41:11.6720275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6720337Z outputs = self.model( 2025-12-04T09:41:11.6720595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6720665Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6720913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6720980Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6721190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6721270Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6721500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6721565Z return func(*args, **kwargs) 2025-12-04T09:41:11.6721811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6721902Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6722138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6722200Z return func(*args, **kwargs) 2025-12-04T09:41:11.6722453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6722545Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6722550Z 2025-12-04T09:41:11.6722626Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6722730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6722945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6723019Z res = mod(**inputs) 2025-12-04T09:41:11.6723275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6723337Z outputs = self.model( 2025-12-04T09:41:11.6723584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6723661Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6723910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6723983Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6724220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6724294Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6724529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6724593Z return func(*args, **kwargs) 2025-12-04T09:41:11.6724834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6724930Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6725161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6725231Z return func(*args, **kwargs) 2025-12-04T09:41:11.6725474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6725570Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6725862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6725989Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6725993Z 2025-12-04T09:41:11.6726094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6726283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6726343Z res = mod(**inputs) 2025-12-04T09:41:11.6726596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6726660Z outputs = self.model( 2025-12-04T09:41:11.6726907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6726985Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6727233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6727310Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6727525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6727599Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6727842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6727905Z return func(*args, **kwargs) 2025-12-04T09:41:11.6728175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6728270Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6728506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6728580Z return func(*args, **kwargs) 2025-12-04T09:41:11.6728838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6728932Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6728943Z 2025-12-04T09:41:11.6729041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6729232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6729300Z res = mod(**inputs) 2025-12-04T09:41:11.6729546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6729612Z outputs = self.model( 2025-12-04T09:41:11.6729866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6729953Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6730210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6730280Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6730496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6730578Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6730815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6730878Z return func(*args, **kwargs) 2025-12-04T09:41:11.6731129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T09:41:11.6731205Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6731210Z 2025-12-04T09:41:11.6731484Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6731717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6731790Z res = mod(**inputs) 2025-12-04T09:41:11.6732080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6732154Z outputs = self.model( 2025-12-04T09:41:11.6732443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6732529Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6732815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6732903Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6733138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6733220Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6733486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6733559Z return func(*args, **kwargs) 2025-12-04T09:41:11.6733835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6733948Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6734202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6734294Z return func(*args, **kwargs) 2025-12-04T09:41:11.6734538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6734681Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6734695Z 2025-12-04T09:41:11.6734794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6734998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6735083Z res = mod(**inputs) 2025-12-04T09:41:11.6735336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6735401Z outputs = self.model( 2025-12-04T09:41:11.6735664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6735735Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6736002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6736072Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6736309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6736391Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6736626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6736691Z return func(*args, **kwargs) 2025-12-04T09:41:11.6736944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6737046Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6737297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6737362Z return func(*args, **kwargs) 2025-12-04T09:41:11.6737601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6737686Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6737689Z 2025-12-04T09:41:11.6737789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6737985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6738045Z res = mod(**inputs) 2025-12-04T09:41:11.6738287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6738356Z outputs = self.model( 2025-12-04T09:41:11.6738602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6738672Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6738921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6738991Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6739213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6739285Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6739521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6739593Z return func(*args, **kwargs) 2025-12-04T09:41:11.6739856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6739958Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6740216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6740284Z return func(*args, **kwargs) 2025-12-04T09:41:11.6740537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6740621Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6740624Z 2025-12-04T09:41:11.6740717Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6740841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6741034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6741103Z res = mod(**inputs) 2025-12-04T09:41:11.6741350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6741414Z outputs = self.model( 2025-12-04T09:41:11.6741666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6741736Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6741980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6742073Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6742293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6742374Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6742607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6742674Z return func(*args, **kwargs) 2025-12-04T09:41:11.6742926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6743028Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6743261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6743335Z return func(*args, **kwargs) 2025-12-04T09:41:11.6743579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6743681Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6743969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6744098Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6744102Z 2025-12-04T09:41:11.6744208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6744402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6744470Z res = mod(**inputs) 2025-12-04T09:41:11.6744716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6744783Z outputs = self.model( 2025-12-04T09:41:11.6745037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6745108Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6745357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6745433Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6745648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6745730Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6745980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6746047Z return func(*args, **kwargs) 2025-12-04T09:41:11.6746298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6746401Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6746662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6746749Z return func(*args, **kwargs) 2025-12-04T09:41:11.6746986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6747072Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6747075Z 2025-12-04T09:41:11.6747174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6747360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6747429Z res = mod(**inputs) 2025-12-04T09:41:11.6747669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6747756Z outputs = self.model( 2025-12-04T09:41:11.6747997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6748067Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6748315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6748381Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6748599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6748672Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6748900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6748968Z return func(*args, **kwargs) 2025-12-04T09:41:11.6749207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6749322Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6749325Z 2025-12-04T09:41:11.6749430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6749616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6749684Z res = mod(**inputs) 2025-12-04T09:41:11.6749926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6749990Z outputs = self.model( 2025-12-04T09:41:11.6750242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6750309Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6750551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6750619Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6750831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6750914Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6751145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6751211Z return func(*args, **kwargs) 2025-12-04T09:41:11.6751456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6751585Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6751589Z 2025-12-04T09:41:11.6751696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6751880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6751940Z res = mod(**inputs) 2025-12-04T09:41:11.6752199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6752276Z outputs = self.model( 2025-12-04T09:41:11.6752513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6752589Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6752827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6752900Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6753115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6753186Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6753422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6753502Z return func(*args, **kwargs) 2025-12-04T09:41:11.6753748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.6753827Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6753830Z 2025-12-04T09:41:11.6753924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6754116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6754175Z res = mod(**inputs) 2025-12-04T09:41:11.6754415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6754485Z outputs = self.model( 2025-12-04T09:41:11.6754725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6754801Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6755040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6755106Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6755324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6755398Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6755629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6755700Z return func(*args, **kwargs) 2025-12-04T09:41:11.6755941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6756040Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6756271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6756335Z return func(*args, **kwargs) 2025-12-04T09:41:11.6756586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6756724Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6756728Z 2025-12-04T09:41:11.6756830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6757015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6757075Z res = mod(**inputs) 2025-12-04T09:41:11.6757341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6757405Z outputs = self.model( 2025-12-04T09:41:11.6757651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6757727Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6757980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6758072Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6758290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6758364Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6758603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6758667Z return func(*args, **kwargs) 2025-12-04T09:41:11.6758913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6759024Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6759252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6759323Z return func(*args, **kwargs) 2025-12-04T09:41:11.6759559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6759633Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6759643Z 2025-12-04T09:41:11.6759739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6759923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6759990Z res = mod(**inputs) 2025-12-04T09:41:11.6760226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6760288Z outputs = self.model( 2025-12-04T09:41:11.6760540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6760609Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6760856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6760933Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6761135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6761212Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6761434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6761498Z return func(*args, **kwargs) 2025-12-04T09:41:11.6761742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6761834Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6762067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6762131Z return func(*args, **kwargs) 2025-12-04T09:41:11.6762366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6762453Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6762457Z 2025-12-04T09:41:11.6762533Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6762636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6762840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6762901Z res = mod(**inputs) 2025-12-04T09:41:11.6763147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6763210Z outputs = self.model( 2025-12-04T09:41:11.6763473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6763562Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6763802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6763875Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6764087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6764159Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6764392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6764454Z return func(*args, **kwargs) 2025-12-04T09:41:11.6764698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6764795Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6765016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6765086Z return func(*args, **kwargs) 2025-12-04T09:41:11.6765322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6765413Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6765696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6765821Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6765824Z 2025-12-04T09:41:11.6765925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6766109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6766171Z res = mod(**inputs) 2025-12-04T09:41:11.6766418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6766481Z outputs = self.model( 2025-12-04T09:41:11.6766718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6766789Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6767035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6767107Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6767314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6767386Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6767617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6767680Z return func(*args, **kwargs) 2025-12-04T09:41:11.6767913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6768011Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6768248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6768319Z return func(*args, **kwargs) 2025-12-04T09:41:11.6768578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6768657Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6768663Z 2025-12-04T09:41:11.6768766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6768974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6769058Z res = mod(**inputs) 2025-12-04T09:41:11.6769307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6769370Z outputs = self.model( 2025-12-04T09:41:11.6769626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6769694Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6769950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6770025Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6770243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6770343Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6770575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6770640Z return func(*args, **kwargs) 2025-12-04T09:41:11.6770885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6770986Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6771223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6771353Z return func(*args, **kwargs) 2025-12-04T09:41:11.6771608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6771760Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6771766Z 2025-12-04T09:41:11.6771865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6772062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6772134Z res = mod(**inputs) 2025-12-04T09:41:11.6772545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6772627Z outputs = self.model( 2025-12-04T09:41:11.6772897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6772978Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6773261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6773331Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6773564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6773642Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6773882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6773957Z return func(*args, **kwargs) 2025-12-04T09:41:11.6774204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6774308Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6774608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6774675Z return func(*args, **kwargs) 2025-12-04T09:41:11.6774931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6775012Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6775016Z 2025-12-04T09:41:11.6775141Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6775362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6775427Z res = mod(**inputs) 2025-12-04T09:41:11.6775716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6775785Z outputs = self.model( 2025-12-04T09:41:11.6776039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6776121Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6776373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6776464Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6776689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6776764Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6777011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6777077Z return func(*args, **kwargs) 2025-12-04T09:41:11.6777329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6777438Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6777684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6777750Z return func(*args, **kwargs) 2025-12-04T09:41:11.6778012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6778096Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6778099Z 2025-12-04T09:41:11.6778185Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6778287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6778482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6778552Z res = mod(**inputs) 2025-12-04T09:41:11.6778809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6778881Z outputs = self.model( 2025-12-04T09:41:11.6779136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6779205Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6779468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6779539Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6779760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6779858Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6780097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6780168Z return func(*args, **kwargs) 2025-12-04T09:41:11.6780416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6780534Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6780776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6780841Z return func(*args, **kwargs) 2025-12-04T09:41:11.6781086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6781203Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6781501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6781636Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6781639Z 2025-12-04T09:41:11.6781740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6781927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6781998Z res = mod(**inputs) 2025-12-04T09:41:11.6782244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6782314Z outputs = self.model( 2025-12-04T09:41:11.6782586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6782656Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6782909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6782977Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6783188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6783268Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6783503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6783576Z return func(*args, **kwargs) 2025-12-04T09:41:11.6783815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6783918Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6784157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6784222Z return func(*args, **kwargs) 2025-12-04T09:41:11.6784470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6784546Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6784549Z 2025-12-04T09:41:11.6784648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6784846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6784906Z res = mod(**inputs) 2025-12-04T09:41:11.6785151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6785222Z outputs = self.model( 2025-12-04T09:41:11.6785466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6785545Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6785788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6785855Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6786077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6786150Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6786401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6786468Z return func(*args, **kwargs) 2025-12-04T09:41:11.6786716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T09:41:11.6786802Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6786805Z 2025-12-04T09:41:11.6786917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6787118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6787187Z res = mod(**inputs) 2025-12-04T09:41:11.6787429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6787501Z outputs = self.model( 2025-12-04T09:41:11.6787747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6787816Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6788065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6788151Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6788369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6788451Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6788685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6788752Z return func(*args, **kwargs) 2025-12-04T09:41:11.6788985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6789094Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6789098Z 2025-12-04T09:41:11.6789201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6789380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6789448Z res = mod(**inputs) 2025-12-04T09:41:11.6789681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6789743Z outputs = self.model( 2025-12-04T09:41:11.6789981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6790047Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6790280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6790353Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6790560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6790637Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6790859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6790923Z return func(*args, **kwargs) 2025-12-04T09:41:11.6791165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6791271Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6791275Z 2025-12-04T09:41:11.6791377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6791560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6791618Z res = mod(**inputs) 2025-12-04T09:41:11.6791892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6791956Z outputs = self.model( 2025-12-04T09:41:11.6792203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6792278Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6792527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6792614Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6792818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6792889Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6793120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6793180Z return func(*args, **kwargs) 2025-12-04T09:41:11.6793414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.6793494Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6793515Z 2025-12-04T09:41:11.6793609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6793797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6793854Z res = mod(**inputs) 2025-12-04T09:41:11.6794088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6794159Z outputs = self.model( 2025-12-04T09:41:11.6794389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6794459Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6794697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6794761Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6794971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6795042Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6795271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6795341Z return func(*args, **kwargs) 2025-12-04T09:41:11.6795571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6795666Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6795897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6795962Z return func(*args, **kwargs) 2025-12-04T09:41:11.6796203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6796339Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6796344Z 2025-12-04T09:41:11.6796445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6796627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6796685Z res = mod(**inputs) 2025-12-04T09:41:11.6796937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6796996Z outputs = self.model( 2025-12-04T09:41:11.6797228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6797302Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6797552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6797624Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6797831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6797917Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6798146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6798224Z return func(*args, **kwargs) 2025-12-04T09:41:11.6798461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6798553Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6798775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6798845Z return func(*args, **kwargs) 2025-12-04T09:41:11.6799077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6799168Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6799171Z 2025-12-04T09:41:11.6799275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6799456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6799522Z res = mod(**inputs) 2025-12-04T09:41:11.6799753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6799814Z outputs = self.model( 2025-12-04T09:41:11.6800052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6800122Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6800353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6800427Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6800633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6800711Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6800933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6800994Z return func(*args, **kwargs) 2025-12-04T09:41:11.6801232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6801324Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6801551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6801614Z return func(*args, **kwargs) 2025-12-04T09:41:11.6801842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6801931Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6801934Z 2025-12-04T09:41:11.6802010Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6802107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6802296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6802355Z res = mod(**inputs) 2025-12-04T09:41:11.6802591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6802652Z outputs = self.model( 2025-12-04T09:41:11.6802897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6802974Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6803214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6803280Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6803511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6803595Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6803823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6803890Z return func(*args, **kwargs) 2025-12-04T09:41:11.6804119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6804217Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6804438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6804507Z return func(*args, **kwargs) 2025-12-04T09:41:11.6804757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6804847Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6805118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6805239Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6805242Z 2025-12-04T09:41:11.6805341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6805521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6805580Z res = mod(**inputs) 2025-12-04T09:41:11.6805826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6805885Z outputs = self.model( 2025-12-04T09:41:11.6806167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6806246Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6806488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6806563Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6806773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6806844Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6807081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6807144Z return func(*args, **kwargs) 2025-12-04T09:41:11.6807379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6807477Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6807703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6807774Z return func(*args, **kwargs) 2025-12-04T09:41:11.6808011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6808085Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6808088Z 2025-12-04T09:41:11.6808192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6808389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6808458Z res = mod(**inputs) 2025-12-04T09:41:11.6808711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6808775Z outputs = self.model( 2025-12-04T09:41:11.6809035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6809104Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6809360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6809435Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6809649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6809726Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6809955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6810018Z return func(*args, **kwargs) 2025-12-04T09:41:11.6810261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6810386Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6810618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6810690Z return func(*args, **kwargs) 2025-12-04T09:41:11.6810933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6811082Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6811085Z 2025-12-04T09:41:11.6811184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6811440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6811515Z res = mod(**inputs) 2025-12-04T09:41:11.6811763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6811837Z outputs = self.model( 2025-12-04T09:41:11.6812092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6812167Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6812428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6812501Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6812726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6812813Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6813056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6813131Z return func(*args, **kwargs) 2025-12-04T09:41:11.6813387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6813494Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6813749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6813819Z return func(*args, **kwargs) 2025-12-04T09:41:11.6814078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6814158Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6814162Z 2025-12-04T09:41:11.6814299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6814504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6814575Z res = mod(**inputs) 2025-12-04T09:41:11.6814816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6814887Z outputs = self.model( 2025-12-04T09:41:11.6815143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6815239Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6815479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6815548Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6815765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6815838Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6816074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6816155Z return func(*args, **kwargs) 2025-12-04T09:41:11.6816395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6816502Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6816733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6816794Z return func(*args, **kwargs) 2025-12-04T09:41:11.6817040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6817121Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6817124Z 2025-12-04T09:41:11.6817206Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6817307Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6817491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6817559Z res = mod(**inputs) 2025-12-04T09:41:11.6817801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6817865Z outputs = self.model( 2025-12-04T09:41:11.6818114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6818183Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6818431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6818495Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6818715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6818795Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6819028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6819100Z return func(*args, **kwargs) 2025-12-04T09:41:11.6819340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6819439Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6819674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6819738Z return func(*args, **kwargs) 2025-12-04T09:41:11.6819975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6820094Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6820373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6820506Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6820509Z 2025-12-04T09:41:11.6820620Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6820822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6820892Z res = mod(**inputs) 2025-12-04T09:41:11.6821144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6821214Z outputs = self.model( 2025-12-04T09:41:11.6821460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6821534Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6821778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6821874Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6822092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6822174Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6822412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6822481Z return func(*args, **kwargs) 2025-12-04T09:41:11.6822727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6822826Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6823068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6823132Z return func(*args, **kwargs) 2025-12-04T09:41:11.6823375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6823458Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6823462Z 2025-12-04T09:41:11.6823559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6823757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6823815Z res = mod(**inputs) 2025-12-04T09:41:11.6824061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6824129Z outputs = self.model( 2025-12-04T09:41:11.6824377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6824452Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6824699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6824766Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6824987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6825062Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6825295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6825366Z return func(*args, **kwargs) 2025-12-04T09:41:11.6825612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6825730Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6825750Z 2025-12-04T09:41:11.6825847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6826031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6826100Z res = mod(**inputs) 2025-12-04T09:41:11.6826344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6826427Z outputs = self.model( 2025-12-04T09:41:11.6826682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6826750Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6826997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6827063Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6827273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6827353Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6827580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6827667Z return func(*args, **kwargs) 2025-12-04T09:41:11.6827906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6828017Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6828021Z 2025-12-04T09:41:11.6828126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6828320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6828388Z res = mod(**inputs) 2025-12-04T09:41:11.6828629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6828691Z outputs = self.model( 2025-12-04T09:41:11.6828938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6829008Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6829247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6829323Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6829533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6829613Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6829838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6829902Z return func(*args, **kwargs) 2025-12-04T09:41:11.6830148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.6830223Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6830226Z 2025-12-04T09:41:11.6830331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6830517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6830579Z res = mod(**inputs) 2025-12-04T09:41:11.6830827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6830889Z outputs = self.model( 2025-12-04T09:41:11.6831125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6831201Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6831456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6831530Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6831739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6831815Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6832066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6832156Z return func(*args, **kwargs) 2025-12-04T09:41:11.6832391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T09:41:11.6832475Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6832479Z 2025-12-04T09:41:11.6832574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6832764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6832823Z res = mod(**inputs) 2025-12-04T09:41:11.6833061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6833148Z outputs = self.model( 2025-12-04T09:41:11.6833390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6833463Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6833700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6833767Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6833984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6834056Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6834286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6834357Z return func(*args, **kwargs) 2025-12-04T09:41:11.6834591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6834693Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6834922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6834987Z return func(*args, **kwargs) 2025-12-04T09:41:11.6835230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6835369Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6835372Z 2025-12-04T09:41:11.6835475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6835661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6835720Z res = mod(**inputs) 2025-12-04T09:41:11.6835967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6836031Z outputs = self.model( 2025-12-04T09:41:11.6836271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6836346Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6836583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6836657Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6836864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6836953Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6837187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6837249Z return func(*args, **kwargs) 2025-12-04T09:41:11.6837487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6837600Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6837842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6837911Z return func(*args, **kwargs) 2025-12-04T09:41:11.6838147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6838222Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6838225Z 2025-12-04T09:41:11.6838328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6838514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6838578Z res = mod(**inputs) 2025-12-04T09:41:11.6838816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6838906Z outputs = self.model( 2025-12-04T09:41:11.6839153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6839221Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6839458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6839531Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6839741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6839820Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6840055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6840118Z return func(*args, **kwargs) 2025-12-04T09:41:11.6840373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6840465Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6840697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6840769Z return func(*args, **kwargs) 2025-12-04T09:41:11.6841015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6841101Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6841105Z 2025-12-04T09:41:11.6841181Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6841275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6841463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6841522Z res = mod(**inputs) 2025-12-04T09:41:11.6841770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6841832Z outputs = self.model( 2025-12-04T09:41:11.6842069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6842143Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6842381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6842450Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6842684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6842757Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6842986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6843053Z return func(*args, **kwargs) 2025-12-04T09:41:11.6843299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6843411Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6843635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6843706Z return func(*args, **kwargs) 2025-12-04T09:41:11.6843939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6844028Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6844309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6844449Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6844453Z 2025-12-04T09:41:11.6844546Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6844736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6844797Z res = mod(**inputs) 2025-12-04T09:41:11.6845047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6845111Z outputs = self.model( 2025-12-04T09:41:11.6845350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6845429Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6845670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6845744Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6845959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6846035Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6846275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6846340Z return func(*args, **kwargs) 2025-12-04T09:41:11.6846578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6846676Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6846907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6846977Z return func(*args, **kwargs) 2025-12-04T09:41:11.6847218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6847293Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6847297Z 2025-12-04T09:41:11.6847402Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6847589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6847649Z res = mod(**inputs) 2025-12-04T09:41:11.6847895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6847958Z outputs = self.model( 2025-12-04T09:41:11.6848224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6848294Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6848535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6848613Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6848841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6848924Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6849165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6849229Z return func(*args, **kwargs) 2025-12-04T09:41:11.6849475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6849576Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6849801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6849870Z return func(*args, **kwargs) 2025-12-04T09:41:11.6850108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6850282Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6850285Z 2025-12-04T09:41:11.6850383Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6850568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6850634Z res = mod(**inputs) 2025-12-04T09:41:11.6850872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6850940Z outputs = self.model( 2025-12-04T09:41:11.6851180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6851245Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6851600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6851682Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6851919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6852012Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6852272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6852351Z return func(*args, **kwargs) 2025-12-04T09:41:11.6852614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6852729Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6852992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6853054Z return func(*args, **kwargs) 2025-12-04T09:41:11.6853294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6853379Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6853384Z 2025-12-04T09:41:11.6853482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6853673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6853733Z res = mod(**inputs) 2025-12-04T09:41:11.6854009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6854080Z outputs = self.model( 2025-12-04T09:41:11.6854330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6854406Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6854645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6854714Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6854958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6855045Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6855273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6855342Z return func(*args, **kwargs) 2025-12-04T09:41:11.6855580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6855686Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6855911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6855991Z return func(*args, **kwargs) 2025-12-04T09:41:11.6856236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6856317Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6856321Z 2025-12-04T09:41:11.6856404Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6856500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6856683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6856748Z res = mod(**inputs) 2025-12-04T09:41:11.6856993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6857055Z outputs = self.model( 2025-12-04T09:41:11.6857303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6857372Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6857622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6857690Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6857898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6857977Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6858206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6858271Z return func(*args, **kwargs) 2025-12-04T09:41:11.6858516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6858613Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6858846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6858909Z return func(*args, **kwargs) 2025-12-04T09:41:11.6859148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6859249Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6859523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6859657Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6859660Z 2025-12-04T09:41:11.6859772Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6859962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6860029Z res = mod(**inputs) 2025-12-04T09:41:11.6860270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6860334Z outputs = self.model( 2025-12-04T09:41:11.6860603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6860687Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6860942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6861010Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6861222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6861304Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6861534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6861623Z return func(*args, **kwargs) 2025-12-04T09:41:11.6861868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6861970Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6862212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6862275Z return func(*args, **kwargs) 2025-12-04T09:41:11.6862521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6862605Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6862608Z 2025-12-04T09:41:11.6862705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6862901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6862961Z res = mod(**inputs) 2025-12-04T09:41:11.6863213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6863286Z outputs = self.model( 2025-12-04T09:41:11.6863538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6863613Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6863862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6863928Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6864153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6864225Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6864460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6864534Z return func(*args, **kwargs) 2025-12-04T09:41:11.6864783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6864904Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6864908Z 2025-12-04T09:41:11.6865005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6865195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6865262Z res = mod(**inputs) 2025-12-04T09:41:11.6865510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6865593Z outputs = self.model( 2025-12-04T09:41:11.6865841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6865910Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6866172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6866240Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6866467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6866547Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6866773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6866843Z return func(*args, **kwargs) 2025-12-04T09:41:11.6867078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6867187Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6867190Z 2025-12-04T09:41:11.6867292Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6867502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6867563Z res = mod(**inputs) 2025-12-04T09:41:11.6867808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6867870Z outputs = self.model( 2025-12-04T09:41:11.6868115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6868182Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6868420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6868492Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6868699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6868781Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6869018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6869081Z return func(*args, **kwargs) 2025-12-04T09:41:11.6869319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.6869391Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6869394Z 2025-12-04T09:41:11.6869487Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6869677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6869736Z res = mod(**inputs) 2025-12-04T09:41:11.6869975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6870036Z outputs = self.model( 2025-12-04T09:41:11.6870267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6870341Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6870576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6870639Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6870851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6870922Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6871168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6871233Z return func(*args, **kwargs) 2025-12-04T09:41:11.6871463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6871566Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6871799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6871885Z return func(*args, **kwargs) 2025-12-04T09:41:11.6872122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6872381Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6872387Z 2025-12-04T09:41:11.6872499Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6872685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6872746Z res = mod(**inputs) 2025-12-04T09:41:11.6872988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6873090Z outputs = self.model( 2025-12-04T09:41:11.6873336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6873405Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6873641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6873717Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6873929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6874006Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6874231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6874294Z return func(*args, **kwargs) 2025-12-04T09:41:11.6874535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6874630Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6874855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6874927Z return func(*args, **kwargs) 2025-12-04T09:41:11.6875158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6875240Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6875243Z 2025-12-04T09:41:11.6875334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6875515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6875581Z res = mod(**inputs) 2025-12-04T09:41:11.6875814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6875882Z outputs = self.model( 2025-12-04T09:41:11.6876116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6876187Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6876428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6876491Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6876694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6876809Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6877033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6877101Z return func(*args, **kwargs) 2025-12-04T09:41:11.6877334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6877443Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6877692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6877754Z return func(*args, **kwargs) 2025-12-04T09:41:11.6877982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6878068Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6878071Z 2025-12-04T09:41:11.6878147Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6878246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6878427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6878504Z res = mod(**inputs) 2025-12-04T09:41:11.6878748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6878811Z outputs = self.model( 2025-12-04T09:41:11.6879052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6879117Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6879349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6879420Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6879624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6879696Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6879926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6879990Z return func(*args, **kwargs) 2025-12-04T09:41:11.6880229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6880318Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6880541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6880609Z return func(*args, **kwargs) 2025-12-04T09:41:11.6880839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6880937Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6881207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6881329Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6881335Z 2025-12-04T09:41:11.6881437Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6881625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6881685Z res = mod(**inputs) 2025-12-04T09:41:11.6881935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6881997Z outputs = self.model( 2025-12-04T09:41:11.6882247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6882333Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6882609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6882682Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6882888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6882987Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6883230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6883292Z return func(*args, **kwargs) 2025-12-04T09:41:11.6883530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6883619Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6883843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6883911Z return func(*args, **kwargs) 2025-12-04T09:41:11.6884143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6884240Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6884244Z 2025-12-04T09:41:11.6884340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6884523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6884588Z res = mod(**inputs) 2025-12-04T09:41:11.6884821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6884881Z outputs = self.model( 2025-12-04T09:41:11.6885121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6885188Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6885429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6885497Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6885707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6885787Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6886011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6886082Z return func(*args, **kwargs) 2025-12-04T09:41:11.6886323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T09:41:11.6886398Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6886401Z 2025-12-04T09:41:11.6886506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6886691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6886749Z res = mod(**inputs) 2025-12-04T09:41:11.6886999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6887060Z outputs = self.model( 2025-12-04T09:41:11.6887308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6887376Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6887617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6887690Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6887916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6887988Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6888262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6888327Z return func(*args, **kwargs) 2025-12-04T09:41:11.6888584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6888700Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6888928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6888999Z return func(*args, **kwargs) 2025-12-04T09:41:11.6889234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6889382Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6889385Z 2025-12-04T09:41:11.6889479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6889665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6889747Z res = mod(**inputs) 2025-12-04T09:41:11.6889994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6890059Z outputs = self.model( 2025-12-04T09:41:11.6890306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6890372Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6890622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6890689Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6890901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6890985Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6891241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6891375Z return func(*args, **kwargs) 2025-12-04T09:41:11.6891659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6891774Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6892036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6892106Z return func(*args, **kwargs) 2025-12-04T09:41:11.6892370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6892455Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6892459Z 2025-12-04T09:41:11.6892559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6892756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6892820Z res = mod(**inputs) 2025-12-04T09:41:11.6893073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6893145Z outputs = self.model( 2025-12-04T09:41:11.6893382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6893457Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6893697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6893765Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6894000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6894075Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6894303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6894373Z return func(*args, **kwargs) 2025-12-04T09:41:11.6894626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6894746Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6894979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6895040Z return func(*args, **kwargs) 2025-12-04T09:41:11.6895289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6895370Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6895373Z 2025-12-04T09:41:11.6895453Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6895546Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6895754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6895820Z res = mod(**inputs) 2025-12-04T09:41:11.6896058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6896120Z outputs = self.model( 2025-12-04T09:41:11.6896364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6896431Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6896679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6896745Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6896952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6897033Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6897262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6897325Z return func(*args, **kwargs) 2025-12-04T09:41:11.6897568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6897666Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6897899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6897962Z return func(*args, **kwargs) 2025-12-04T09:41:11.6898199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6898298Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6898575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6898708Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6898712Z 2025-12-04T09:41:11.6898810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6898995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6899062Z res = mod(**inputs) 2025-12-04T09:41:11.6899303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6899366Z outputs = self.model( 2025-12-04T09:41:11.6899631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6899700Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6899947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6900017Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6900243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6900336Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6900566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6900629Z return func(*args, **kwargs) 2025-12-04T09:41:11.6900871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6900970Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6901203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6901297Z return func(*args, **kwargs) 2025-12-04T09:41:11.6901536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6901622Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6901625Z 2025-12-04T09:41:11.6901721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6901913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6901973Z res = mod(**inputs) 2025-12-04T09:41:11.6902214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6902285Z outputs = self.model( 2025-12-04T09:41:11.6902522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6902589Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6902835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6902903Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6903118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6903188Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6903416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6903488Z return func(*args, **kwargs) 2025-12-04T09:41:11.6903723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6903841Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6903845Z 2025-12-04T09:41:11.6903940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6904125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6904189Z res = mod(**inputs) 2025-12-04T09:41:11.6904428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6904491Z outputs = self.model( 2025-12-04T09:41:11.6904735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6904802Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6905064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6905132Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6905344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6905426Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6905669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6905740Z return func(*args, **kwargs) 2025-12-04T09:41:11.6905991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6906100Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6906104Z 2025-12-04T09:41:11.6906206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6906388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6906447Z res = mod(**inputs) 2025-12-04T09:41:11.6906693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6906756Z outputs = self.model( 2025-12-04T09:41:11.6907016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6907084Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6907325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6907398Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6907604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6907673Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6907908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6907971Z return func(*args, **kwargs) 2025-12-04T09:41:11.6908212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.6908288Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6908292Z 2025-12-04T09:41:11.6908388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6908578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6908637Z res = mod(**inputs) 2025-12-04T09:41:11.6908879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6908941Z outputs = self.model( 2025-12-04T09:41:11.6909178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6909253Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6909492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6909563Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6909778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6909860Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6910087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6910147Z return func(*args, **kwargs) 2025-12-04T09:41:11.6910377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6910474Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6910723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6910795Z return func(*args, **kwargs) 2025-12-04T09:41:11.6911030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6911169Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6911187Z 2025-12-04T09:41:11.6911290Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6911483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6911540Z res = mod(**inputs) 2025-12-04T09:41:11.6911778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6911838Z outputs = self.model( 2025-12-04T09:41:11.6912080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6912147Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6912379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6912467Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6912673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6912745Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6912975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6913035Z return func(*args, **kwargs) 2025-12-04T09:41:11.6913273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6913364Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6913586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6913655Z return func(*args, **kwargs) 2025-12-04T09:41:11.6913888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6913971Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6913975Z 2025-12-04T09:41:11.6914069Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6914252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6914315Z res = mod(**inputs) 2025-12-04T09:41:11.6914546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6914607Z outputs = self.model( 2025-12-04T09:41:11.6914848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6914914Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6915150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6915217Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6915420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6915499Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6915721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6915787Z return func(*args, **kwargs) 2025-12-04T09:41:11.6916016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6916117Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6916346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6916407Z return func(*args, **kwargs) 2025-12-04T09:41:11.6916639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6916738Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6916763Z 2025-12-04T09:41:11.6916839Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6916940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6917123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6917180Z res = mod(**inputs) 2025-12-04T09:41:11.6917418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6917481Z outputs = self.model( 2025-12-04T09:41:11.6917715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6917805Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6918043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6918116Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6918320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6918390Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6918618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6918679Z return func(*args, **kwargs) 2025-12-04T09:41:11.6918920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6919010Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6919232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6919302Z return func(*args, **kwargs) 2025-12-04T09:41:11.6919535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6919624Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6919902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6920022Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6920026Z 2025-12-04T09:41:11.6920126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6920308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6920365Z res = mod(**inputs) 2025-12-04T09:41:11.6920606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6920668Z outputs = self.model( 2025-12-04T09:41:11.6920908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6920974Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6921208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6921281Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6921486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6921571Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6921802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6921866Z return func(*args, **kwargs) 2025-12-04T09:41:11.6922104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6922205Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6922440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6922509Z return func(*args, **kwargs) 2025-12-04T09:41:11.6922740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6922820Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6922823Z 2025-12-04T09:41:11.6922918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6923097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6923160Z res = mod(**inputs) 2025-12-04T09:41:11.6923391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6923482Z outputs = self.model( 2025-12-04T09:41:11.6923734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6923801Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6924039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6924103Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6924306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6924381Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6924603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6924665Z return func(*args, **kwargs) 2025-12-04T09:41:11.6924902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6925001Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6925229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6925290Z return func(*args, **kwargs) 2025-12-04T09:41:11.6925519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6925661Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6925666Z 2025-12-04T09:41:11.6925760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6925946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6926005Z res = mod(**inputs) 2025-12-04T09:41:11.6926237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6926305Z outputs = self.model( 2025-12-04T09:41:11.6926541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6926607Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6926850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6926916Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6927141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6927215Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6927438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6927509Z return func(*args, **kwargs) 2025-12-04T09:41:11.6927754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6927874Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6928102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6928162Z return func(*args, **kwargs) 2025-12-04T09:41:11.6928403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6928478Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6928481Z 2025-12-04T09:41:11.6928574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6928762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6928838Z res = mod(**inputs) 2025-12-04T09:41:11.6929085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6929149Z outputs = self.model( 2025-12-04T09:41:11.6929391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6929466Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6929703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6929775Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6929983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6930053Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6930289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6930354Z return func(*args, **kwargs) 2025-12-04T09:41:11.6930591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6930699Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6930927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6931001Z return func(*args, **kwargs) 2025-12-04T09:41:11.6931395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.6931511Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.6931515Z 2025-12-04T09:41:11.6931610Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.6931721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6931939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6932017Z res = mod(**inputs) 2025-12-04T09:41:11.6932300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6932382Z outputs = self.model( 2025-12-04T09:41:11.6932653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6932722Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6933000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6933067Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6933284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6933359Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6933603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6933688Z return func(*args, **kwargs) 2025-12-04T09:41:11.6933930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6934030Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6934270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6934333Z return func(*args, **kwargs) 2025-12-04T09:41:11.6934591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.6934681Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.6934956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.6935107Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.6935111Z 2025-12-04T09:41:11.6935208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6935401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6935460Z res = mod(**inputs) 2025-12-04T09:41:11.6935699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6935771Z outputs = self.model( 2025-12-04T09:41:11.6936010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6936078Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6936321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6936389Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6936604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6936677Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6936904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6936971Z return func(*args, **kwargs) 2025-12-04T09:41:11.6937215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.6937316Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.6937549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6937615Z return func(*args, **kwargs) 2025-12-04T09:41:11.6937861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.6937937Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.6937941Z 2025-12-04T09:41:11.6938036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6938230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6938289Z res = mod(**inputs) 2025-12-04T09:41:11.6938532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6938596Z outputs = self.model( 2025-12-04T09:41:11.6938847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6938924Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6939175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6939266Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6939503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6939577Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6939814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6939877Z return func(*args, **kwargs) 2025-12-04T09:41:11.6940118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T09:41:11.6940200Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.6940203Z 2025-12-04T09:41:11.6940298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6940507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6940566Z res = mod(**inputs) 2025-12-04T09:41:11.6940808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6940880Z outputs = self.model( 2025-12-04T09:41:11.6941120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6941188Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6941435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6941503Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6941721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6941792Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6942024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6942098Z return func(*args, **kwargs) 2025-12-04T09:41:11.6942338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6942451Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6942462Z 2025-12-04T09:41:11.6942557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6942745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6942810Z res = mod(**inputs) 2025-12-04T09:41:11.6943052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6943115Z outputs = self.model( 2025-12-04T09:41:11.6943360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6943429Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6943675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6943742Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6943950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6944029Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6944273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6944341Z return func(*args, **kwargs) 2025-12-04T09:41:11.6944589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.6944702Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.6944705Z 2025-12-04T09:41:11.6944805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6945001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6945076Z res = mod(**inputs) 2025-12-04T09:41:11.6945326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6945388Z outputs = self.model( 2025-12-04T09:41:11.6945632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6945699Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6945934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6946007Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6946234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6946306Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6946552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6946613Z return func(*args, **kwargs) 2025-12-04T09:41:11.6946851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.6946926Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.6946929Z 2025-12-04T09:41:11.6947026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6947214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6947271Z res = mod(**inputs) 2025-12-04T09:41:11.6947512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6947575Z outputs = self.model( 2025-12-04T09:41:11.6947808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6947881Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6948114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6948177Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6948386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6948456Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6948684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6948746Z return func(*args, **kwargs) 2025-12-04T09:41:11.6948976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6949075Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6949297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6949358Z return func(*args, **kwargs) 2025-12-04T09:41:11.6949594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.6949729Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.6949749Z 2025-12-04T09:41:11.6949853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6950032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6994496Z res = mod(**inputs) 2025-12-04T09:41:11.6995082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6995419Z outputs = self.model( 2025-12-04T09:41:11.6995773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6995856Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6996120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6996196Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6996430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6996516Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.6996760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6996887Z return func(*args, **kwargs) 2025-12-04T09:41:11.6997135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.6997243Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.6997535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.6997629Z return func(*args, **kwargs) 2025-12-04T09:41:11.6997910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.6997997Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.6998005Z 2025-12-04T09:41:11.6998116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.6998327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.6998396Z res = mod(**inputs) 2025-12-04T09:41:11.6998660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.6998736Z outputs = self.model( 2025-12-04T09:41:11.6998992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.6999083Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.6999356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.6999435Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.6999667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.6999750Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7000003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7000076Z return func(*args, **kwargs) 2025-12-04T09:41:11.7000328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7000441Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7000677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7000751Z return func(*args, **kwargs) 2025-12-04T09:41:11.7001000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7001134Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7001142Z 2025-12-04T09:41:11.7001233Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7001337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7001543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7001617Z res = mod(**inputs) 2025-12-04T09:41:11.7001914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7002017Z outputs = self.model( 2025-12-04T09:41:11.7002269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7002342Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7002596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7002667Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7002880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7002989Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7003224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7003308Z return func(*args, **kwargs) 2025-12-04T09:41:11.7003549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7003642Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7003871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7003935Z return func(*args, **kwargs) 2025-12-04T09:41:11.7004180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7004275Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7004550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7004687Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7004693Z 2025-12-04T09:41:11.7004792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7004982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7005052Z res = mod(**inputs) 2025-12-04T09:41:11.7005294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7005364Z outputs = self.model( 2025-12-04T09:41:11.7005622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7005694Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7005945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7006016Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7006239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7006317Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7006547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7006622Z return func(*args, **kwargs) 2025-12-04T09:41:11.7006864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7006976Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7007218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7007284Z return func(*args, **kwargs) 2025-12-04T09:41:11.7007533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7007626Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7007644Z 2025-12-04T09:41:11.7007749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7007950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7008015Z res = mod(**inputs) 2025-12-04T09:41:11.7008259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7008333Z outputs = self.model( 2025-12-04T09:41:11.7008607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7008687Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7008929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7009019Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7009252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7009334Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7009590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7009668Z return func(*args, **kwargs) 2025-12-04T09:41:11.7010000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7010161Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7017373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7017592Z return func(*args, **kwargs) 2025-12-04T09:41:11.7017915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7018076Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7018082Z 2025-12-04T09:41:11.7018205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7018415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7018486Z res = mod(**inputs) 2025-12-04T09:41:11.7039653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7039781Z outputs = self.model( 2025-12-04T09:41:11.7040061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7040148Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7040388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7040459Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7040674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7040750Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7040988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7041053Z return func(*args, **kwargs) 2025-12-04T09:41:11.7041398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7041509Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7041740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7041811Z return func(*args, **kwargs) 2025-12-04T09:41:11.7042067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7042167Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7042174Z 2025-12-04T09:41:11.7042281Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7042512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7042580Z res = mod(**inputs) 2025-12-04T09:41:11.7042823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7042886Z outputs = self.model( 2025-12-04T09:41:11.7043123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7043237Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7043474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7043543Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7043750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7043822Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7044043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7044104Z return func(*args, **kwargs) 2025-12-04T09:41:11.7044338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7044437Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7044663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7044722Z return func(*args, **kwargs) 2025-12-04T09:41:11.7044952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7045034Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7045039Z 2025-12-04T09:41:11.7045114Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7045210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7045398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7045457Z res = mod(**inputs) 2025-12-04T09:41:11.7045693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7045753Z outputs = self.model( 2025-12-04T09:41:11.7045986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7046056Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7046285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7046351Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7046553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7046623Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7046860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7046922Z return func(*args, **kwargs) 2025-12-04T09:41:11.7047152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7047254Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7047489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7047569Z return func(*args, **kwargs) 2025-12-04T09:41:11.7047812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7047903Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7048188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7048319Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7048323Z 2025-12-04T09:41:11.7048422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7048610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7048687Z res = mod(**inputs) 2025-12-04T09:41:11.7048933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7048995Z outputs = self.model( 2025-12-04T09:41:11.7049234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7049304Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7049547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7049615Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7049825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7049896Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7050128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7050190Z return func(*args, **kwargs) 2025-12-04T09:41:11.7050471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7050574Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7050803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7050867Z return func(*args, **kwargs) 2025-12-04T09:41:11.7051116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7051198Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7051202Z 2025-12-04T09:41:11.7051400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7051615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7051688Z res = mod(**inputs) 2025-12-04T09:41:11.7051963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7052033Z outputs = self.model( 2025-12-04T09:41:11.7052308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7052394Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7052637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7052726Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7052935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7053008Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7053232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7053308Z return func(*args, **kwargs) 2025-12-04T09:41:11.7053550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7053681Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7053685Z 2025-12-04T09:41:11.7053783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7053968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7054024Z res = mod(**inputs) 2025-12-04T09:41:11.7054273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7054339Z outputs = self.model( 2025-12-04T09:41:11.7054585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7054678Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7054919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7054987Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7055201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7055274Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7055521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7055585Z return func(*args, **kwargs) 2025-12-04T09:41:11.7055815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7055931Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7055935Z 2025-12-04T09:41:11.7056030Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7056226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7056286Z res = mod(**inputs) 2025-12-04T09:41:11.7056525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7056595Z outputs = self.model( 2025-12-04T09:41:11.7056840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7056906Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7057148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7057212Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7057425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7057498Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7057721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7057793Z return func(*args, **kwargs) 2025-12-04T09:41:11.7058024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.7058099Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.7058109Z 2025-12-04T09:41:11.7058203Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7058399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7058467Z res = mod(**inputs) 2025-12-04T09:41:11.7058703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7058767Z outputs = self.model( 2025-12-04T09:41:11.7059022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7059104Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7059381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7059447Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7059657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7059738Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7059966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7060029Z return func(*args, **kwargs) 2025-12-04T09:41:11.7060288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T09:41:11.7060365Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.7060369Z 2025-12-04T09:41:11.7060473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7060658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7060717Z res = mod(**inputs) 2025-12-04T09:41:11.7060958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7061019Z outputs = self.model( 2025-12-04T09:41:11.7061260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7061326Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7061558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7061634Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7061841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7061914Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7062145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7062209Z return func(*args, **kwargs) 2025-12-04T09:41:11.7062448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7062544Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7062764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7062838Z return func(*args, **kwargs) 2025-12-04T09:41:11.7063071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7063216Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7063227Z 2025-12-04T09:41:11.7063321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7063503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7063569Z res = mod(**inputs) 2025-12-04T09:41:11.7063803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7063880Z outputs = self.model( 2025-12-04T09:41:11.7064125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7064195Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7064493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7064562Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7064799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7064878Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7065099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7065162Z return func(*args, **kwargs) 2025-12-04T09:41:11.7065401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7065494Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7065720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7065803Z return func(*args, **kwargs) 2025-12-04T09:41:11.7066040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7066126Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7066129Z 2025-12-04T09:41:11.7066222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7066409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7066468Z res = mod(**inputs) 2025-12-04T09:41:11.7066703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7066771Z outputs = self.model( 2025-12-04T09:41:11.7067006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7067076Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7067319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7067386Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7067598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7067668Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7067889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7067956Z return func(*args, **kwargs) 2025-12-04T09:41:11.7068189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7068282Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7068503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7068565Z return func(*args, **kwargs) 2025-12-04T09:41:11.7068804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7068883Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7068886Z 2025-12-04T09:41:11.7068960Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7069060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7069240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7069305Z res = mod(**inputs) 2025-12-04T09:41:11.7069555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7069619Z outputs = self.model( 2025-12-04T09:41:11.7069859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7069927Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7070172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7070258Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7070467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7070545Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7070771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7070834Z return func(*args, **kwargs) 2025-12-04T09:41:11.7071076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7071182Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7071411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7071474Z return func(*args, **kwargs) 2025-12-04T09:41:11.7071705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7071803Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7072076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7072201Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7072211Z 2025-12-04T09:41:11.7072464Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7072655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7072724Z res = mod(**inputs) 2025-12-04T09:41:11.7072963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7073023Z outputs = self.model( 2025-12-04T09:41:11.7073263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7073328Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7073569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7073634Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7073842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7073921Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7074145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7074211Z return func(*args, **kwargs) 2025-12-04T09:41:11.7074456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7074545Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7074779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7074843Z return func(*args, **kwargs) 2025-12-04T09:41:11.7075078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7075235Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7075239Z 2025-12-04T09:41:11.7075332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7075519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7075577Z res = mod(**inputs) 2025-12-04T09:41:11.7075835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7075929Z outputs = self.model( 2025-12-04T09:41:11.7076163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7076231Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7076471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7076535Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7076749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7076819Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7077061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7077129Z return func(*args, **kwargs) 2025-12-04T09:41:11.7077362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7077462Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7077684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7077744Z return func(*args, **kwargs) 2025-12-04T09:41:11.7077980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7078119Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7078123Z 2025-12-04T09:41:11.7078218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7078406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7078466Z res = mod(**inputs) 2025-12-04T09:41:11.7078709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7078773Z outputs = self.model( 2025-12-04T09:41:11.7079006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7079082Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7079318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7079383Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7079593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7079667Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7079895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7079957Z return func(*args, **kwargs) 2025-12-04T09:41:11.7080191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7080299Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7080521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7080589Z return func(*args, **kwargs) 2025-12-04T09:41:11.7080837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7080911Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7080914Z 2025-12-04T09:41:11.7081017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7081199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7081271Z res = mod(**inputs) 2025-12-04T09:41:11.7081528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7081588Z outputs = self.model( 2025-12-04T09:41:11.7081827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7081893Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7082127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7082199Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7082402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7082488Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7082714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7082779Z return func(*args, **kwargs) 2025-12-04T09:41:11.7083014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7083110Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7083328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7083393Z return func(*args, **kwargs) 2025-12-04T09:41:11.7083624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7083710Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7083715Z 2025-12-04T09:41:11.7083790Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7083885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7084074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7084133Z res = mod(**inputs) 2025-12-04T09:41:11.7084366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7084433Z outputs = self.model( 2025-12-04T09:41:11.7084665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7084740Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7084976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7085042Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7085257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7085329Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7085557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7085620Z return func(*args, **kwargs) 2025-12-04T09:41:11.7085849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7085953Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7086232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7086298Z return func(*args, **kwargs) 2025-12-04T09:41:11.7086544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7086638Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7086946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7087090Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7087093Z 2025-12-04T09:41:11.7087191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7087382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7087442Z res = mod(**inputs) 2025-12-04T09:41:11.7087696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7087759Z outputs = self.model( 2025-12-04T09:41:11.7087999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7088091Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7088336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7088406Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7088628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7088702Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7088941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7089007Z return func(*args, **kwargs) 2025-12-04T09:41:11.7089247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7089354Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7089586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7089651Z return func(*args, **kwargs) 2025-12-04T09:41:11.7089901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7089980Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7089984Z 2025-12-04T09:41:11.7090087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7090276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7090336Z res = mod(**inputs) 2025-12-04T09:41:11.7090596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7090664Z outputs = self.model( 2025-12-04T09:41:11.7090934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7091003Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7091246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7091428Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7091669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7091752Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7092018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7092114Z return func(*args, **kwargs) 2025-12-04T09:41:11.7092390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7092516Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7092523Z 2025-12-04T09:41:11.7092628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7092853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7092929Z res = mod(**inputs) 2025-12-04T09:41:11.7093186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7093250Z outputs = self.model( 2025-12-04T09:41:11.7093544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7093617Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7093861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7093929Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7094148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7094237Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7094477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7094542Z return func(*args, **kwargs) 2025-12-04T09:41:11.7094782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7094900Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7094903Z 2025-12-04T09:41:11.7095000Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7095196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7095255Z res = mod(**inputs) 2025-12-04T09:41:11.7095498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7095570Z outputs = self.model( 2025-12-04T09:41:11.7095815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7095883Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7096133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7096200Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7096418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7096491Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7096719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7096790Z return func(*args, **kwargs) 2025-12-04T09:41:11.7097030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.7097108Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.7097119Z 2025-12-04T09:41:11.7097218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7097401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7097467Z res = mod(**inputs) 2025-12-04T09:41:11.7097709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7097772Z outputs = self.model( 2025-12-04T09:41:11.7098034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7098101Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7098343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7098410Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7098634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7098727Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7098972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7099036Z return func(*args, **kwargs) 2025-12-04T09:41:11.7099286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7099380Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7099611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7099699Z return func(*args, **kwargs) 2025-12-04T09:41:11.7099937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7100088Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7100093Z 2025-12-04T09:41:11.7100189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7100373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7100439Z res = mod(**inputs) 2025-12-04T09:41:11.7100678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7100748Z outputs = self.model( 2025-12-04T09:41:11.7100987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7101056Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7101303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7101371Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7101583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7101662Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7101886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7101954Z return func(*args, **kwargs) 2025-12-04T09:41:11.7102194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7102283Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7102512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7102575Z return func(*args, **kwargs) 2025-12-04T09:41:11.7102970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7103057Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7103060Z 2025-12-04T09:41:11.7103154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7103349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7103409Z res = mod(**inputs) 2025-12-04T09:41:11.7103669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7103746Z outputs = self.model( 2025-12-04T09:41:11.7103983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7104061Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7104316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7104399Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7104617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7104691Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7104921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7104992Z return func(*args, **kwargs) 2025-12-04T09:41:11.7105232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7105331Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7105559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7105646Z return func(*args, **kwargs) 2025-12-04T09:41:11.7105897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7106323Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7106465Z 2025-12-04T09:41:11.7106548Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7106797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7107138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7107433Z res = mod(**inputs) 2025-12-04T09:41:11.7107775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7108138Z outputs = self.model( 2025-12-04T09:41:11.7108481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7108854Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7109221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7109593Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7109921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7110266Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7110628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7111007Z return func(*args, **kwargs) 2025-12-04T09:41:11.7111432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7111818Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7112194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7112545Z return func(*args, **kwargs) 2025-12-04T09:41:11.7112899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7113279Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7113704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7114168Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7114374Z 2025-12-04T09:41:11.7114472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7114807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7115111Z res = mod(**inputs) 2025-12-04T09:41:11.7115468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7115832Z outputs = self.model( 2025-12-04T09:41:11.7116199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7116565Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7116921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7117283Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7117614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7117958Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7118316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7118699Z return func(*args, **kwargs) 2025-12-04T09:41:11.7119070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7119477Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7119870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7120223Z return func(*args, **kwargs) 2025-12-04T09:41:11.7120580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7120952Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7121088Z 2025-12-04T09:41:11.7121184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7121530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7121847Z res = mod(**inputs) 2025-12-04T09:41:11.7122193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7122567Z outputs = self.model( 2025-12-04T09:41:11.7122922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7123314Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7123673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7124054Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7124387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7124739Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7125108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7125475Z return func(*args, **kwargs) 2025-12-04T09:41:11.7125844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T09:41:11.7126222Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.7126361Z 2025-12-04T09:41:11.7126458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7126809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7127119Z res = mod(**inputs) 2025-12-04T09:41:11.7127492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7127873Z outputs = self.model( 2025-12-04T09:41:11.7128232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7128616Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7129010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7129413Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7129767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7130124Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7130500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7130870Z return func(*args, **kwargs) 2025-12-04T09:41:11.7131325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7131791Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7132260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7132637Z return func(*args, **kwargs) 2025-12-04T09:41:11.7133011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7133479Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7133694Z 2025-12-04T09:41:11.7133805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7134153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7134462Z res = mod(**inputs) 2025-12-04T09:41:11.7134823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7135202Z outputs = self.model( 2025-12-04T09:41:11.7135557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7135939Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7136315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7136693Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7137031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7137388Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7137758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7138127Z return func(*args, **kwargs) 2025-12-04T09:41:11.7138493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7138903Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7139302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7139658Z return func(*args, **kwargs) 2025-12-04T09:41:11.7140022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7140410Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7140541Z 2025-12-04T09:41:11.7140647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7140990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7141318Z res = mod(**inputs) 2025-12-04T09:41:11.7141681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7142044Z outputs = self.model( 2025-12-04T09:41:11.7142402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7142799Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7143198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7143588Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7143926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7144277Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7144642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7145005Z return func(*args, **kwargs) 2025-12-04T09:41:11.7145363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7145783Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7146164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7146515Z return func(*args, **kwargs) 2025-12-04T09:41:11.7146868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7147244Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7147378Z 2025-12-04T09:41:11.7147452Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7147679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7148020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7148318Z res = mod(**inputs) 2025-12-04T09:41:11.7148657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7149018Z outputs = self.model( 2025-12-04T09:41:11.7149366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7149728Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7150088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7150453Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7150777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7151122Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7151480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7151833Z return func(*args, **kwargs) 2025-12-04T09:41:11.7152236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7152641Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7153025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7153373Z return func(*args, **kwargs) 2025-12-04T09:41:11.7153719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7154101Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7154543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7155000Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7155181Z 2025-12-04T09:41:11.7155279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7155619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7155942Z res = mod(**inputs) 2025-12-04T09:41:11.7156348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7156712Z outputs = self.model( 2025-12-04T09:41:11.7157065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7157433Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7157794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7158167Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7158498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7158869Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7159229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7159586Z return func(*args, **kwargs) 2025-12-04T09:41:11.7159940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7160331Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7160715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7161064Z return func(*args, **kwargs) 2025-12-04T09:41:11.7161411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7161797Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7161933Z 2025-12-04T09:41:11.7162031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7162368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7162670Z res = mod(**inputs) 2025-12-04T09:41:11.7163014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7163373Z outputs = self.model( 2025-12-04T09:41:11.7163712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7164074Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7164428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7164791Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7165105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7165446Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7165793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7166139Z return func(*args, **kwargs) 2025-12-04T09:41:11.7166483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7166893Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7167054Z 2025-12-04T09:41:11.7167159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7167517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7167815Z res = mod(**inputs) 2025-12-04T09:41:11.7168159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7168522Z outputs = self.model( 2025-12-04T09:41:11.7168875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7169253Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7169613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7169978Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7170298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7170639Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7171002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7171486Z return func(*args, **kwargs) 2025-12-04T09:41:11.7171904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7172571Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7172744Z 2025-12-04T09:41:11.7172855Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7173202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7173515Z res = mod(**inputs) 2025-12-04T09:41:11.7173872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7174249Z outputs = self.model( 2025-12-04T09:41:11.7174602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7174988Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7175367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7175750Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7176090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7176437Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7176800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7177152Z return func(*args, **kwargs) 2025-12-04T09:41:11.7177511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.7177894Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.7178028Z 2025-12-04T09:41:11.7178129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7178471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7178778Z res = mod(**inputs) 2025-12-04T09:41:11.7179126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7179488Z outputs = self.model( 2025-12-04T09:41:11.7179835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7180206Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7180561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7180930Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7181300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7181648Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7181998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7182354Z return func(*args, **kwargs) 2025-12-04T09:41:11.7182732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7183155Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7183535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7183890Z return func(*args, **kwargs) 2025-12-04T09:41:11.7184242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7184688Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7184890Z 2025-12-04T09:41:11.7184990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7185357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7185660Z res = mod(**inputs) 2025-12-04T09:41:11.7185994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7186358Z outputs = self.model( 2025-12-04T09:41:11.7186706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7187073Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7187425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7187793Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7188123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7188459Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7188818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7189172Z return func(*args, **kwargs) 2025-12-04T09:41:11.7189524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7189907Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7190288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7190641Z return func(*args, **kwargs) 2025-12-04T09:41:11.7190988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7191361Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7191494Z 2025-12-04T09:41:11.7191591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7191929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7192228Z res = mod(**inputs) 2025-12-04T09:41:11.7192568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7192933Z outputs = self.model( 2025-12-04T09:41:11.7193283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7193646Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7194036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7194413Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7194743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7195093Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7195478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7195860Z return func(*args, **kwargs) 2025-12-04T09:41:11.7196208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7196598Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7196974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7197320Z return func(*args, **kwargs) 2025-12-04T09:41:11.7197682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7198062Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7198194Z 2025-12-04T09:41:11.7198298Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7198522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7198867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7199184Z res = mod(**inputs) 2025-12-04T09:41:11.7199527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7199886Z outputs = self.model( 2025-12-04T09:41:11.7200239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7200611Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7200971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7201344Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7201675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7202028Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7202386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7202745Z return func(*args, **kwargs) 2025-12-04T09:41:11.7203099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7203485Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7203867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7204222Z return func(*args, **kwargs) 2025-12-04T09:41:11.7204577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7204964Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7205394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7205860Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7206036Z 2025-12-04T09:41:11.7206142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7206481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7206789Z res = mod(**inputs) 2025-12-04T09:41:11.7207150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7207509Z outputs = self.model( 2025-12-04T09:41:11.7207855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7208228Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7208608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7208996Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7209327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7209673Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7210033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7210383Z return func(*args, **kwargs) 2025-12-04T09:41:11.7210738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7211131Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7211622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7212051Z return func(*args, **kwargs) 2025-12-04T09:41:11.7212453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7212878Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7213025Z 2025-12-04T09:41:11.7213137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7213527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7213849Z res = mod(**inputs) 2025-12-04T09:41:11.7214196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7214575Z outputs = self.model( 2025-12-04T09:41:11.7214970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7215384Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7215783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7216195Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7216564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7216945Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7217338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7217731Z return func(*args, **kwargs) 2025-12-04T09:41:11.7218128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7218569Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7219009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7219401Z return func(*args, **kwargs) 2025-12-04T09:41:11.7219797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7220280Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7220506Z 2025-12-04T09:41:11.7220615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7220992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7221329Z res = mod(**inputs) 2025-12-04T09:41:11.7221730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7222099Z outputs = self.model( 2025-12-04T09:41:11.7222448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7222809Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7223187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7223573Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7223924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7224274Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7224642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7225028Z return func(*args, **kwargs) 2025-12-04T09:41:11.7225377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7225800Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7226193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7226554Z return func(*args, **kwargs) 2025-12-04T09:41:11.7226906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7227283Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7227410Z 2025-12-04T09:41:11.7227514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7227860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7228161Z res = mod(**inputs) 2025-12-04T09:41:11.7228506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7228882Z outputs = self.model( 2025-12-04T09:41:11.7229229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7229613Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7229986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7230357Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7230684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7231027Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7231391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7231742Z return func(*args, **kwargs) 2025-12-04T09:41:11.7232097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7232498Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7232889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7233244Z return func(*args, **kwargs) 2025-12-04T09:41:11.7233598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7233977Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7234110Z 2025-12-04T09:41:11.7234195Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7234415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7234784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7235099Z res = mod(**inputs) 2025-12-04T09:41:11.7235441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7235814Z outputs = self.model( 2025-12-04T09:41:11.7236185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7236577Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7236944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7237309Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7237648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7237988Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7238349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7238699Z return func(*args, **kwargs) 2025-12-04T09:41:11.7239058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7239436Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7239816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7240158Z return func(*args, **kwargs) 2025-12-04T09:41:11.7240500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7240871Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7241284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7241728Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7241898Z 2025-12-04T09:41:11.7241996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7242328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7242628Z res = mod(**inputs) 2025-12-04T09:41:11.7242962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7243306Z outputs = self.model( 2025-12-04T09:41:11.7243643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7244000Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7244348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7244707Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7245031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7245367Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7245713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7246059Z return func(*args, **kwargs) 2025-12-04T09:41:11.7246405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7246793Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7247163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7247503Z return func(*args, **kwargs) 2025-12-04T09:41:11.7247865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7248228Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7248367Z 2025-12-04T09:41:11.7248463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7248808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7249113Z res = mod(**inputs) 2025-12-04T09:41:11.7249464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7249827Z outputs = self.model( 2025-12-04T09:41:11.7250169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7250533Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7250886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7251303Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7251654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7252027Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7252405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7252768Z return func(*args, **kwargs) 2025-12-04T09:41:11.7253133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T09:41:11.7253516Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.7253659Z 2025-12-04T09:41:11.7253763Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7254120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7254429Z res = mod(**inputs) 2025-12-04T09:41:11.7254791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7255172Z outputs = self.model( 2025-12-04T09:41:11.7255540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7255910Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7256281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7256656Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7256985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7257336Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7257705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7258067Z return func(*args, **kwargs) 2025-12-04T09:41:11.7258419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7258846Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7259022Z 2025-12-04T09:41:11.7259124Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7259469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7259772Z res = mod(**inputs) 2025-12-04T09:41:11.7260122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7260493Z outputs = self.model( 2025-12-04T09:41:11.7260857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7261240Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7261616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7261992Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7262347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7262723Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7263093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7263444Z return func(*args, **kwargs) 2025-12-04T09:41:11.7263780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7264178Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7264335Z 2025-12-04T09:41:11.7264438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7264762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7265081Z res = mod(**inputs) 2025-12-04T09:41:11.7265414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7265771Z outputs = self.model( 2025-12-04T09:41:11.7266100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7266458Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7266814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7267168Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7267493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7267823Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7268174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7268513Z return func(*args, **kwargs) 2025-12-04T09:41:11.7268857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.7269223Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.7269346Z 2025-12-04T09:41:11.7269450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7269769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7270063Z res = mod(**inputs) 2025-12-04T09:41:11.7270395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7270740Z outputs = self.model( 2025-12-04T09:41:11.7271086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7271451Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7271813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7272173Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7272614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7272964Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7273319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7273683Z return func(*args, **kwargs) 2025-12-04T09:41:11.7274089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7274473Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7274843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7275188Z return func(*args, **kwargs) 2025-12-04T09:41:11.7275554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7276012Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7276202Z 2025-12-04T09:41:11.7276299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7276632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7276932Z res = mod(**inputs) 2025-12-04T09:41:11.7277263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7277617Z outputs = self.model( 2025-12-04T09:41:11.7277956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7278340Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7278687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7279048Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7279373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7279710Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7280055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7280399Z return func(*args, **kwargs) 2025-12-04T09:41:11.7280740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7281114Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7281482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7281822Z return func(*args, **kwargs) 2025-12-04T09:41:11.7282169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7282523Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7282653Z 2025-12-04T09:41:11.7282750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7283077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7283373Z res = mod(**inputs) 2025-12-04T09:41:11.7283717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7284082Z outputs = self.model( 2025-12-04T09:41:11.7284422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7284785Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7285032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7285101Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7285314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7285396Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7285642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7285719Z return func(*args, **kwargs) 2025-12-04T09:41:11.7285957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7286052Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7286306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7286386Z return func(*args, **kwargs) 2025-12-04T09:41:11.7286634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7286716Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7286720Z 2025-12-04T09:41:11.7286795Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7286901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7287092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7287151Z res = mod(**inputs) 2025-12-04T09:41:11.7287405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7287484Z outputs = self.model( 2025-12-04T09:41:11.7287734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7287804Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7288042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7288117Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7288326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7288399Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7288636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7288699Z return func(*args, **kwargs) 2025-12-04T09:41:11.7288943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7289038Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7289267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7289338Z return func(*args, **kwargs) 2025-12-04T09:41:11.7289576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7289677Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7289955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7290079Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7290083Z 2025-12-04T09:41:11.7290187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7290373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7290439Z res = mod(**inputs) 2025-12-04T09:41:11.7290682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7290745Z outputs = self.model( 2025-12-04T09:41:11.7290997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7291066Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7291379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7291467Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7291699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7291789Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7292043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7292130Z return func(*args, **kwargs) 2025-12-04T09:41:11.7292393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T09:41:11.7292491Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:11.7292733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7292809Z return func(*args, **kwargs) 2025-12-04T09:41:11.7293062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7293151Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7293155Z 2025-12-04T09:41:11.7293276Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7293478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7293552Z res = mod(**inputs) 2025-12-04T09:41:11.7293796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7293870Z outputs = self.model( 2025-12-04T09:41:11.7294114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7294184Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7294435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7294503Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7294714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7294798Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7295030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7295103Z return func(*args, **kwargs) 2025-12-04T09:41:11.7295341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7295444Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7295680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7295743Z return func(*args, **kwargs) 2025-12-04T09:41:11.7295983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T09:41:11.7296135Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:11.7296140Z 2025-12-04T09:41:11.7296236Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7296431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7296493Z res = mod(**inputs) 2025-12-04T09:41:11.7296738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7296809Z outputs = self.model( 2025-12-04T09:41:11.7297050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7297125Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7297394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7297464Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7297684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7297760Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7298006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7298093Z return func(*args, **kwargs) 2025-12-04T09:41:11.7298341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7298450Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7298681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7298744Z return func(*args, **kwargs) 2025-12-04T09:41:11.7298988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T09:41:11.7299085Z key_states = self.k_proj(current_states) 2025-12-04T09:41:11.7299089Z 2025-12-04T09:41:11.7299194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7299383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7299443Z res = mod(**inputs) 2025-12-04T09:41:11.7299690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7299751Z outputs = self.model( 2025-12-04T09:41:11.7299988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7300063Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7300299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7300373Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7300585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7300658Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7300897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7300959Z return func(*args, **kwargs) 2025-12-04T09:41:11.7301199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7301297Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7301525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7301596Z return func(*args, **kwargs) 2025-12-04T09:41:11.7301830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T09:41:11.7301913Z value_states = self.v_proj(current_states) 2025-12-04T09:41:11.7301916Z 2025-12-04T09:41:11.7302002Z cudagraph partition due to non gpu ops 2025-12-04T09:41:11.7302099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7302290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7302350Z res = mod(**inputs) 2025-12-04T09:41:11.7302587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7302657Z outputs = self.model( 2025-12-04T09:41:11.7302922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7302992Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7303237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7303306Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7303538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7303628Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7303868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7303936Z return func(*args, **kwargs) 2025-12-04T09:41:11.7304171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7304277Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7304498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7304559Z return func(*args, **kwargs) 2025-12-04T09:41:11.7304818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T09:41:11.7304911Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:11.7305184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:11.7305315Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:11.7305318Z 2025-12-04T09:41:11.7305411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7305599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7305658Z res = mod(**inputs) 2025-12-04T09:41:11.7305892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7305960Z outputs = self.model( 2025-12-04T09:41:11.7306195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7306272Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7306508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7306573Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7306784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7306855Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7307079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7307150Z return func(*args, **kwargs) 2025-12-04T09:41:11.7307382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T09:41:11.7307489Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T09:41:11.7307712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7307778Z return func(*args, **kwargs) 2025-12-04T09:41:11.7308017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T09:41:11.7308091Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:11.7308094Z 2025-12-04T09:41:11.7308195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7308393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7308455Z res = mod(**inputs) 2025-12-04T09:41:11.7308701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7308765Z outputs = self.model( 2025-12-04T09:41:11.7309019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7309096Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7309340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7309411Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7309622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7309691Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7309918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7309978Z return func(*args, **kwargs) 2025-12-04T09:41:11.7310206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7310340Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7310343Z 2025-12-04T09:41:11.7310439Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7310630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7310687Z res = mod(**inputs) 2025-12-04T09:41:11.7310920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7310990Z outputs = self.model( 2025-12-04T09:41:11.7311228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7311301Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7311534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7311600Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7311813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7311886Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7312110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7312180Z return func(*args, **kwargs) 2025-12-04T09:41:11.7312411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T09:41:11.7312526Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:11.7312531Z 2025-12-04T09:41:11.7312625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7312804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7312875Z res = mod(**inputs) 2025-12-04T09:41:11.7313110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7313178Z outputs = self.model( 2025-12-04T09:41:11.7313414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7313477Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7313720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7313784Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7314011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7314092Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7314315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7314387Z return func(*args, **kwargs) 2025-12-04T09:41:11.7314632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T09:41:11.7314721Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:11.7314726Z 2025-12-04T09:41:11.7314826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7315005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7315063Z res = mod(**inputs) 2025-12-04T09:41:11.7315301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T09:41:11.7315360Z outputs = self.model( 2025-12-04T09:41:11.7315599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T09:41:11.7315689Z decoder_outputs = self.decoder( 2025-12-04T09:41:11.7315934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T09:41:11.7316007Z layer_outputs = decoder_layer( 2025-12-04T09:41:11.7316215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:11.7316292Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:11.7316525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:11.7316585Z return func(*args, **kwargs) 2025-12-04T09:41:11.7316837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T09:41:11.7316909Z hidden_states = residual + hidden_states 2025-12-04T09:41:11.7316912Z 2025-12-04T09:41:11.7317007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7317200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7317257Z res = mod(**inputs) 2025-12-04T09:41:11.7317509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1419, in forward 2025-12-04T09:41:11.7317581Z lm_logits = self.lm_head(outputs[0]) 2025-12-04T09:41:11.7317584Z 2025-12-04T09:41:11.7317678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:11.7317868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:11.7317925Z res = mod(**inputs) 2025-12-04T09:41:11.7318175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1426, in forward 2025-12-04T09:41:11.7318332Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:41:11.7318337Z 2025-12-04T09:41:23.0818645Z Compilation time (from dynamo_timed): 26.097325312 2025-12-04T09:41:23.0923898Z pass 2025-12-04T09:41:23.0924537Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:23.0925429Z TIMING: _recursive_pre_grad_passes:0.01361 _recursive_joint_graph_passes:1.09697 _recursive_post_grad_passes:0.12105 async_compile.wait:0.73051 code_gen:11.16311 inductor_compile:14.13928 backend_compile:20.54351 gc:0.00062 entire_frame_compile:26.09733 total_wall_time:26.09733 2025-12-04T09:41:23.0927008Z STATS: call_* op count: 906 | FakeTensorMode.__torch_dispatch__:18114 | FakeTensor.__torch_dispatch__:10222 | ProxyTorchDispatchMode.__torch_dispatch__:4990 2025-12-04T09:41:23.0927945Z Dynamo produced 1 graphs covering 906 ops with 0 graph breaks (0 unique) 2025-12-04T09:41:25.9000620Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:41:25.9003289Z import pynvml # type: ignore[import] 2025-12-04T09:41:29.2291359Z 2025-12-04T09:41:31.5120033Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:41:31.5120329Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:41:31.5144116Z cpu eval MBartForCausalLM 2025-12-04T09:41:33.2446084Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:33.9110401Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:34.5901128Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:41.5837996Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5840986Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5841653Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5841873Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5842093Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5842303Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5842513Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5842717Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5842913Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5843117Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5843320Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5843515Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5843758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5844157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5844503Z res = mod(**inputs) 2025-12-04T09:41:41.5844934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5845387Z outputs = self.model.decoder( 2025-12-04T09:41:41.5845934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5846368Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5846756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5847158Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5847584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5847988Z return func(*args, **kwargs) 2025-12-04T09:41:41.5848428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5848887Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5849325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5849721Z return func(*args, **kwargs) 2025-12-04T09:41:41.5850123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.5850623Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.5850860Z 2025-12-04T09:41:41.5850982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5851462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5851831Z res = mod(**inputs) 2025-12-04T09:41:41.5852518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5852941Z outputs = self.model.decoder( 2025-12-04T09:41:41.5853327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5853718Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5854130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5854550Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5854952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5855358Z return func(*args, **kwargs) 2025-12-04T09:41:41.5855754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5856215Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5856673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5857092Z return func(*args, **kwargs) 2025-12-04T09:41:41.5857462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.5857865Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.5858013Z 2025-12-04T09:41:41.5858117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5858487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5858807Z res = mod(**inputs) 2025-12-04T09:41:41.5859177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5859576Z outputs = self.model.decoder( 2025-12-04T09:41:41.5859959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5860351Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5860716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5861081Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5861460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5861837Z return func(*args, **kwargs) 2025-12-04T09:41:41.5862211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5862627Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5863024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5863397Z return func(*args, **kwargs) 2025-12-04T09:41:41.5863771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.5864165Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.5864314Z 2025-12-04T09:41:41.5864397Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5864652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5865009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5865324Z res = mod(**inputs) 2025-12-04T09:41:41.5865690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5866081Z outputs = self.model.decoder( 2025-12-04T09:41:41.5866478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5866870Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5867219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5867584Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5867975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5868364Z return func(*args, **kwargs) 2025-12-04T09:41:41.5868736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5869143Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5869534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5869914Z return func(*args, **kwargs) 2025-12-04T09:41:41.5870291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.5870696Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.5871166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.5871668Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.5871852Z 2025-12-04T09:41:41.5871964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5872521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5872858Z res = mod(**inputs) 2025-12-04T09:41:41.5873222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5873618Z outputs = self.model.decoder( 2025-12-04T09:41:41.5874004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5874394Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5874749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5875116Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5875500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5875873Z return func(*args, **kwargs) 2025-12-04T09:41:41.5876278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5876685Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5877077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5877442Z return func(*args, **kwargs) 2025-12-04T09:41:41.5877816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.5878218Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.5878352Z 2025-12-04T09:41:41.5878458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5878797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5879109Z res = mod(**inputs) 2025-12-04T09:41:41.5879461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5879837Z outputs = self.model.decoder( 2025-12-04T09:41:41.5880208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5880584Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5880969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5881321Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5881694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5882063Z return func(*args, **kwargs) 2025-12-04T09:41:41.5882492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.5882938Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.5883111Z 2025-12-04T09:41:41.5883212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5883567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5883874Z res = mod(**inputs) 2025-12-04T09:41:41.5884233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5884617Z outputs = self.model.decoder( 2025-12-04T09:41:41.5884988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5885415Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5885848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5886212Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5886583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5886949Z return func(*args, **kwargs) 2025-12-04T09:41:41.5887318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.5887748Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.5888127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.5888471Z return self.act(input) 2025-12-04T09:41:41.5888581Z 2025-12-04T09:41:41.5888691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5889042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5889362Z res = mod(**inputs) 2025-12-04T09:41:41.5889721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5890108Z outputs = self.model.decoder( 2025-12-04T09:41:41.5890514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5890934Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5891435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5891841Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5892262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5892678Z return func(*args, **kwargs) 2025-12-04T09:41:41.5893052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.5893444Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.5893590Z 2025-12-04T09:41:41.5893694Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5894059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5894372Z res = mod(**inputs) 2025-12-04T09:41:41.5894749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5895140Z outputs = self.model.decoder( 2025-12-04T09:41:41.5895524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5895909Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5896280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5896663Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5897041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5897404Z return func(*args, **kwargs) 2025-12-04T09:41:41.5897773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5898183Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5898572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5898938Z return func(*args, **kwargs) 2025-12-04T09:41:41.5899307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.5899794Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.5899999Z 2025-12-04T09:41:41.5900100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5900456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5900772Z res = mod(**inputs) 2025-12-04T09:41:41.5901130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5901510Z outputs = self.model.decoder( 2025-12-04T09:41:41.5901891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5902276Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5902615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5902978Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5903359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5903734Z return func(*args, **kwargs) 2025-12-04T09:41:41.5904100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5904513Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5904913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5905282Z return func(*args, **kwargs) 2025-12-04T09:41:41.5905640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.5906015Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.5906145Z 2025-12-04T09:41:41.5906249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5906578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5906882Z res = mod(**inputs) 2025-12-04T09:41:41.5907223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5907590Z outputs = self.model.decoder( 2025-12-04T09:41:41.5907945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5908313Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5908660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5908999Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5909358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5909714Z return func(*args, **kwargs) 2025-12-04T09:41:41.5910082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5910481Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5910859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5911208Z return func(*args, **kwargs) 2025-12-04T09:41:41.5911558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.5911939Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.5912079Z 2025-12-04T09:41:41.5912159Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5912387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5912740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5913043Z res = mod(**inputs) 2025-12-04T09:41:41.5913388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5913756Z outputs = self.model.decoder( 2025-12-04T09:41:41.5914108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5914473Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5914802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5915135Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5915491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5915841Z return func(*args, **kwargs) 2025-12-04T09:41:41.5916194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5916574Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5916950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5917298Z return func(*args, **kwargs) 2025-12-04T09:41:41.5917645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.5918034Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.5918455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.5918912Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.5919086Z 2025-12-04T09:41:41.5919181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5919521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5919826Z res = mod(**inputs) 2025-12-04T09:41:41.5920166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5920527Z outputs = self.model.decoder( 2025-12-04T09:41:41.5920887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5921251Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5921587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5921935Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5922306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5922652Z return func(*args, **kwargs) 2025-12-04T09:41:41.5923007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5923441Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5923806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5924146Z return func(*args, **kwargs) 2025-12-04T09:41:41.5924482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.5924853Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.5924981Z 2025-12-04T09:41:41.5925088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5925416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5925741Z res = mod(**inputs) 2025-12-04T09:41:41.5926086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5926459Z outputs = self.model.decoder( 2025-12-04T09:41:41.5926810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5927172Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5927501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5927832Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5928190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5928543Z return func(*args, **kwargs) 2025-12-04T09:41:41.5928893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.5929293Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.5929463Z 2025-12-04T09:41:41.5929563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5929900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5930202Z res = mod(**inputs) 2025-12-04T09:41:41.5930539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5930912Z outputs = self.model.decoder( 2025-12-04T09:41:41.5931399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5931828Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5932210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5932607Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5933010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5933365Z return func(*args, **kwargs) 2025-12-04T09:41:41.5933731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.5934148Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.5934511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.5934842Z return self.act(input) 2025-12-04T09:41:41.5935016Z 2025-12-04T09:41:41.5935116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5935451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5935745Z res = mod(**inputs) 2025-12-04T09:41:41.5936102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5936474Z outputs = self.model.decoder( 2025-12-04T09:41:41.5936846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5937213Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5937544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5937884Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5938238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5938590Z return func(*args, **kwargs) 2025-12-04T09:41:41.5938943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.5939338Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.5939467Z 2025-12-04T09:41:41.5939567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5939911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5940220Z res = mod(**inputs) 2025-12-04T09:41:41.5940556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5940927Z outputs = self.model.decoder( 2025-12-04T09:41:41.5941294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5941664Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5941994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5942341Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5942703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5943052Z return func(*args, **kwargs) 2025-12-04T09:41:41.5943423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:41:41.5943800Z hidden_states = residual + hidden_states 2025-12-04T09:41:41.5943927Z 2025-12-04T09:41:41.5944031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5944363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5944668Z res = mod(**inputs) 2025-12-04T09:41:41.5945013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5945383Z outputs = self.model.decoder( 2025-12-04T09:41:41.5945743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5946110Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5946444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5946780Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5947142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5947495Z return func(*args, **kwargs) 2025-12-04T09:41:41.5947863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5948249Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5948629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5948981Z return func(*args, **kwargs) 2025-12-04T09:41:41.5949344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.5949799Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.5949998Z 2025-12-04T09:41:41.5950097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5950432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5950725Z res = mod(**inputs) 2025-12-04T09:41:41.5951065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5951434Z outputs = self.model.decoder( 2025-12-04T09:41:41.5951789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5952168Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5952501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5952844Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5953196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5953562Z return func(*args, **kwargs) 2025-12-04T09:41:41.5953905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5954287Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5954649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5954994Z return func(*args, **kwargs) 2025-12-04T09:41:41.5955338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.5955706Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.5955830Z 2025-12-04T09:41:41.5955926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5956258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5956554Z res = mod(**inputs) 2025-12-04T09:41:41.5956880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5957244Z outputs = self.model.decoder( 2025-12-04T09:41:41.5957599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5957957Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5958273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5958608Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5958959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5959302Z return func(*args, **kwargs) 2025-12-04T09:41:41.5959655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5960045Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5960423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5960768Z return func(*args, **kwargs) 2025-12-04T09:41:41.5961135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.5961515Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.5961648Z 2025-12-04T09:41:41.5961733Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.5961950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5962303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5962623Z res = mod(**inputs) 2025-12-04T09:41:41.5962960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5963325Z outputs = self.model.decoder( 2025-12-04T09:41:41.5963692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5964048Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5964367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5964709Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5965063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5965426Z return func(*args, **kwargs) 2025-12-04T09:41:41.5965782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5966175Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5966555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5966900Z return func(*args, **kwargs) 2025-12-04T09:41:41.5967256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.5967648Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.5968067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.5968530Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.5968712Z 2025-12-04T09:41:41.5968813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5969157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5969458Z res = mod(**inputs) 2025-12-04T09:41:41.5969807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5970182Z outputs = self.model.decoder( 2025-12-04T09:41:41.5970548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5970924Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5971334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5971691Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5972093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5972730Z return func(*args, **kwargs) 2025-12-04T09:41:41.5973160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5973610Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5974019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5974398Z return func(*args, **kwargs) 2025-12-04T09:41:41.5974830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.5975226Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.5975362Z 2025-12-04T09:41:41.5975465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5975827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5976178Z res = mod(**inputs) 2025-12-04T09:41:41.5976528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5976947Z outputs = self.model.decoder( 2025-12-04T09:41:41.5977330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5977721Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5978065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5978423Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5978800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5979205Z return func(*args, **kwargs) 2025-12-04T09:41:41.5979597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.5980016Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.5980180Z 2025-12-04T09:41:41.5980284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5980621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5980931Z res = mod(**inputs) 2025-12-04T09:41:41.5981282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5981671Z outputs = self.model.decoder( 2025-12-04T09:41:41.5982052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5982431Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5982770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5983116Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5983490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5983849Z return func(*args, **kwargs) 2025-12-04T09:41:41.5984211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.5984619Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.5984998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.5985341Z return self.act(input) 2025-12-04T09:41:41.5985449Z 2025-12-04T09:41:41.5985551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5985913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5986231Z res = mod(**inputs) 2025-12-04T09:41:41.5986590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5986974Z outputs = self.model.decoder( 2025-12-04T09:41:41.5987362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5987727Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5988056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5988425Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5988807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5989182Z return func(*args, **kwargs) 2025-12-04T09:41:41.5989552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.5989968Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.5990120Z 2025-12-04T09:41:41.5990230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5990588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5990897Z res = mod(**inputs) 2025-12-04T09:41:41.5991257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5991648Z outputs = self.model.decoder( 2025-12-04T09:41:41.5992022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5992412Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5992760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5993144Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5993512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5993886Z return func(*args, **kwargs) 2025-12-04T09:41:41.5994260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.5994663Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.5995064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.5995432Z return func(*args, **kwargs) 2025-12-04T09:41:41.5995803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.5996258Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.5996469Z 2025-12-04T09:41:41.5996573Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.5996931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.5997255Z res = mod(**inputs) 2025-12-04T09:41:41.5997610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.5998001Z outputs = self.model.decoder( 2025-12-04T09:41:41.5998383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.5998764Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.5999123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.5999466Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.5999825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6000170Z return func(*args, **kwargs) 2025-12-04T09:41:41.6000526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6000918Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6001297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6001642Z return func(*args, **kwargs) 2025-12-04T09:41:41.6002016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6002392Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6002519Z 2025-12-04T09:41:41.6002618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6002955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6003258Z res = mod(**inputs) 2025-12-04T09:41:41.6003616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6003989Z outputs = self.model.decoder( 2025-12-04T09:41:41.6004347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6004711Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6005032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6005378Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6005734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6006103Z return func(*args, **kwargs) 2025-12-04T09:41:41.6006452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6006843Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6007223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6007575Z return func(*args, **kwargs) 2025-12-04T09:41:41.6007920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6008299Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6008430Z 2025-12-04T09:41:41.6008517Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6008735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6009073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6009377Z res = mod(**inputs) 2025-12-04T09:41:41.6009720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6010087Z outputs = self.model.decoder( 2025-12-04T09:41:41.6010445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6010809Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6011132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6011590Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6012000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6012397Z return func(*args, **kwargs) 2025-12-04T09:41:41.6012789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6013199Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6013580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6013925Z return func(*args, **kwargs) 2025-12-04T09:41:41.6014280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6014675Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6015119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6015578Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6015761Z 2025-12-04T09:41:41.6015857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6016200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6016506Z res = mod(**inputs) 2025-12-04T09:41:41.6016867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6017264Z outputs = self.model.decoder( 2025-12-04T09:41:41.6017629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6017999Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6018336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6018689Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6019052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6019404Z return func(*args, **kwargs) 2025-12-04T09:41:41.6019793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6020195Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6020582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6020942Z return func(*args, **kwargs) 2025-12-04T09:41:41.6021301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6021683Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6021814Z 2025-12-04T09:41:41.6021914Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6022259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6022568Z res = mod(**inputs) 2025-12-04T09:41:41.6022919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6023290Z outputs = self.model.decoder( 2025-12-04T09:41:41.6023657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6024036Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6024367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6024719Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6025090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6025456Z return func(*args, **kwargs) 2025-12-04T09:41:41.6025807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6026223Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6026391Z 2025-12-04T09:41:41.6026498Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6026842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6027145Z res = mod(**inputs) 2025-12-04T09:41:41.6027494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6027870Z outputs = self.model.decoder( 2025-12-04T09:41:41.6028228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6028621Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6028974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6029330Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6029701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6030088Z return func(*args, **kwargs) 2025-12-04T09:41:41.6030452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6030892Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6031266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6031599Z return self.act(input) 2025-12-04T09:41:41.6031705Z 2025-12-04T09:41:41.6031814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6032151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6032467Z res = mod(**inputs) 2025-12-04T09:41:41.6032805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6033190Z outputs = self.model.decoder( 2025-12-04T09:41:41.6033540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6033909Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6034236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6034564Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6034917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6035266Z return func(*args, **kwargs) 2025-12-04T09:41:41.6035616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6035980Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6036115Z 2025-12-04T09:41:41.6036209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6036541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6036834Z res = mod(**inputs) 2025-12-04T09:41:41.6037169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6037532Z outputs = self.model.decoder( 2025-12-04T09:41:41.6037889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6038244Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6038573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6038912Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6039260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6039613Z return func(*args, **kwargs) 2025-12-04T09:41:41.6039966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:41:41.6040347Z hidden_states = residual + hidden_states 2025-12-04T09:41:41.6040471Z 2025-12-04T09:41:41.6040568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6040901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6041204Z res = mod(**inputs) 2025-12-04T09:41:41.6041557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6041918Z outputs = self.model.decoder( 2025-12-04T09:41:41.6042284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6042656Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6043001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6043369Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6043723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6044072Z return func(*args, **kwargs) 2025-12-04T09:41:41.6044416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6044805Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6045182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6045523Z return func(*args, **kwargs) 2025-12-04T09:41:41.6045875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.6046337Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.6046528Z 2025-12-04T09:41:41.6046631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6046965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6047269Z res = mod(**inputs) 2025-12-04T09:41:41.6047614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6047980Z outputs = self.model.decoder( 2025-12-04T09:41:41.6048335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6048701Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6049029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6049366Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6049727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6050077Z return func(*args, **kwargs) 2025-12-04T09:41:41.6050425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6050808Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6051211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6051713Z return func(*args, **kwargs) 2025-12-04T09:41:41.6052113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6052518Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6052661Z 2025-12-04T09:41:41.6052765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6053125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6053440Z res = mod(**inputs) 2025-12-04T09:41:41.6053807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6054189Z outputs = self.model.decoder( 2025-12-04T09:41:41.6054589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6055001Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6055398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6055781Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6056177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6056574Z return func(*args, **kwargs) 2025-12-04T09:41:41.6056988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6057450Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6057865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6058266Z return func(*args, **kwargs) 2025-12-04T09:41:41.6058661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6059095Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6059246Z 2025-12-04T09:41:41.6059332Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6059585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6059985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6060313Z res = mod(**inputs) 2025-12-04T09:41:41.6060706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6061079Z outputs = self.model.decoder( 2025-12-04T09:41:41.6061443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6061817Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6062158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6062511Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6062873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6063249Z return func(*args, **kwargs) 2025-12-04T09:41:41.6063605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6063995Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6064367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6064717Z return func(*args, **kwargs) 2025-12-04T09:41:41.6065070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6065451Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6065876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6066336Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6066511Z 2025-12-04T09:41:41.6066615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6066949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6067254Z res = mod(**inputs) 2025-12-04T09:41:41.6067596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6067967Z outputs = self.model.decoder( 2025-12-04T09:41:41.6068321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6068687Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6069036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6069381Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6069743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6070097Z return func(*args, **kwargs) 2025-12-04T09:41:41.6070470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6070872Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6071247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6071598Z return func(*args, **kwargs) 2025-12-04T09:41:41.6071950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6072489Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6072630Z 2025-12-04T09:41:41.6072729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6073073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6073431Z res = mod(**inputs) 2025-12-04T09:41:41.6073792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6074176Z outputs = self.model.decoder( 2025-12-04T09:41:41.6074546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6074975Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6075303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6075648Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6075999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6076351Z return func(*args, **kwargs) 2025-12-04T09:41:41.6076707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6077122Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6077288Z 2025-12-04T09:41:41.6077390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6077748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6078057Z res = mod(**inputs) 2025-12-04T09:41:41.6078409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6078783Z outputs = self.model.decoder( 2025-12-04T09:41:41.6079161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6079548Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6079918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6080274Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6080645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6081010Z return func(*args, **kwargs) 2025-12-04T09:41:41.6081362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6081780Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6082154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6082483Z return self.act(input) 2025-12-04T09:41:41.6082625Z 2025-12-04T09:41:41.6082729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6083080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6083393Z res = mod(**inputs) 2025-12-04T09:41:41.6083734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6084175Z outputs = self.model.decoder( 2025-12-04T09:41:41.6084579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6084961Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6085299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6085656Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6086037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6086401Z return func(*args, **kwargs) 2025-12-04T09:41:41.6086772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6087177Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6087316Z 2025-12-04T09:41:41.6087429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6087783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6088106Z res = mod(**inputs) 2025-12-04T09:41:41.6088466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6088850Z outputs = self.model.decoder( 2025-12-04T09:41:41.6089231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6089617Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6089965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6090320Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6090699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6091086Z return func(*args, **kwargs) 2025-12-04T09:41:41.6091515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6091951Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6092386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6092759Z return func(*args, **kwargs) 2025-12-04T09:41:41.6093133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.6093587Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.6093793Z 2025-12-04T09:41:41.6093898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6094253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6094561Z res = mod(**inputs) 2025-12-04T09:41:41.6094917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6095297Z outputs = self.model.decoder( 2025-12-04T09:41:41.6095662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6096041Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6096399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6096750Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6097107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6097476Z return func(*args, **kwargs) 2025-12-04T09:41:41.6097845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6098255Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6098630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6098979Z return func(*args, **kwargs) 2025-12-04T09:41:41.6099328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6099695Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6099831Z 2025-12-04T09:41:41.6099930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6100266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6100585Z res = mod(**inputs) 2025-12-04T09:41:41.6100916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6101282Z outputs = self.model.decoder( 2025-12-04T09:41:41.6101639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6101996Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6102324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6102665Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6103019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6103360Z return func(*args, **kwargs) 2025-12-04T09:41:41.6103711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6104101Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6104474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6104816Z return func(*args, **kwargs) 2025-12-04T09:41:41.6105171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6105548Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6105678Z 2025-12-04T09:41:41.6105755Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6105984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6106323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6106623Z res = mod(**inputs) 2025-12-04T09:41:41.6106958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6107333Z outputs = self.model.decoder( 2025-12-04T09:41:41.6107695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6108058Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6108389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6108729Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6109085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6109448Z return func(*args, **kwargs) 2025-12-04T09:41:41.6109804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6110192Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6110561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6110925Z return func(*args, **kwargs) 2025-12-04T09:41:41.6111297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6111682Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6112095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6112552Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6112730Z 2025-12-04T09:41:41.6112828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6113166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6113474Z res = mod(**inputs) 2025-12-04T09:41:41.6113807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6114168Z outputs = self.model.decoder( 2025-12-04T09:41:41.6114511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6114876Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6115205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6115547Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6115906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6116246Z return func(*args, **kwargs) 2025-12-04T09:41:41.6116590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6116975Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6117336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6117675Z return func(*args, **kwargs) 2025-12-04T09:41:41.6118016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6118382Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6118517Z 2025-12-04T09:41:41.6118615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6118952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6119254Z res = mod(**inputs) 2025-12-04T09:41:41.6119585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6119960Z outputs = self.model.decoder( 2025-12-04T09:41:41.6120313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6120663Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6120987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6121319Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6121665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6121999Z return func(*args, **kwargs) 2025-12-04T09:41:41.6122360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6122770Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6122928Z 2025-12-04T09:41:41.6123030Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6123365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6123679Z res = mod(**inputs) 2025-12-04T09:41:41.6124039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6124404Z outputs = self.model.decoder( 2025-12-04T09:41:41.6124765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6125134Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6125468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6125808Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6126167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6126534Z return func(*args, **kwargs) 2025-12-04T09:41:41.6126880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6127287Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6127650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6127970Z return self.act(input) 2025-12-04T09:41:41.6128072Z 2025-12-04T09:41:41.6128167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6128512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6128823Z res = mod(**inputs) 2025-12-04T09:41:41.6129179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6129614Z outputs = self.model.decoder( 2025-12-04T09:41:41.6129987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6130363Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6130696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6131049Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6131526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6131947Z return func(*args, **kwargs) 2025-12-04T09:41:41.6132341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6132772Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6132919Z 2025-12-04T09:41:41.6133035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6133425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6133749Z res = mod(**inputs) 2025-12-04T09:41:41.6134109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6134502Z outputs = self.model.decoder( 2025-12-04T09:41:41.6134882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6135270Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6135613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6136012Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6136374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6136737Z return func(*args, **kwargs) 2025-12-04T09:41:41.6137100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:41:41.6137527Z hidden_states = residual + hidden_states 2025-12-04T09:41:41.6137685Z 2025-12-04T09:41:41.6137787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6138142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6138458Z res = mod(**inputs) 2025-12-04T09:41:41.6138808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6139191Z outputs = self.model.decoder( 2025-12-04T09:41:41.6139565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6139945Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6140290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6140660Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6141027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6141380Z return func(*args, **kwargs) 2025-12-04T09:41:41.6141741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6142142Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6142518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6142879Z return func(*args, **kwargs) 2025-12-04T09:41:41.6143240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.6143690Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.6143884Z 2025-12-04T09:41:41.6143987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6144344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6144661Z res = mod(**inputs) 2025-12-04T09:41:41.6144999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6145357Z outputs = self.model.decoder( 2025-12-04T09:41:41.6145718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6146084Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6146403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6146747Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6147113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6147468Z return func(*args, **kwargs) 2025-12-04T09:41:41.6147817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6148208Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6148585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6148934Z return func(*args, **kwargs) 2025-12-04T09:41:41.6149330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6149719Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6149846Z 2025-12-04T09:41:41.6149952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6150285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6150589Z res = mod(**inputs) 2025-12-04T09:41:41.6150939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6151321Z outputs = self.model.decoder( 2025-12-04T09:41:41.6151672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6152041Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6152367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6152705Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6153067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6153443Z return func(*args, **kwargs) 2025-12-04T09:41:41.6153794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6154182Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6154564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6154917Z return func(*args, **kwargs) 2025-12-04T09:41:41.6155270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6155642Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6155780Z 2025-12-04T09:41:41.6155858Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6156088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6156418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6156728Z res = mod(**inputs) 2025-12-04T09:41:41.6157084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6157474Z outputs = self.model.decoder( 2025-12-04T09:41:41.6157828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6158204Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6158544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6158895Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6159257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6159611Z return func(*args, **kwargs) 2025-12-04T09:41:41.6159965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6160350Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6160731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6161082Z return func(*args, **kwargs) 2025-12-04T09:41:41.6161427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6161819Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6162246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6162733Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6162913Z 2025-12-04T09:41:41.6163012Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6163353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6163657Z res = mod(**inputs) 2025-12-04T09:41:41.6164014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6164393Z outputs = self.model.decoder( 2025-12-04T09:41:41.6164918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6165295Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6165636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6165981Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6166348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6166708Z return func(*args, **kwargs) 2025-12-04T09:41:41.6167088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6167494Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6167888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6168250Z return func(*args, **kwargs) 2025-12-04T09:41:41.6168607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6168990Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6169122Z 2025-12-04T09:41:41.6169230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6169570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6169881Z res = mod(**inputs) 2025-12-04T09:41:41.6170228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6170609Z outputs = self.model.decoder( 2025-12-04T09:41:41.6170973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6171425Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6171767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6172152Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6172749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6173163Z return func(*args, **kwargs) 2025-12-04T09:41:41.6173556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6173984Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6174171Z 2025-12-04T09:41:41.6174273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6174647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6174964Z res = mod(**inputs) 2025-12-04T09:41:41.6175317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6175699Z outputs = self.model.decoder( 2025-12-04T09:41:41.6176081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6176499Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6176845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6177201Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6177574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6177928Z return func(*args, **kwargs) 2025-12-04T09:41:41.6178316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6178765Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6179150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6179474Z return self.act(input) 2025-12-04T09:41:41.6179588Z 2025-12-04T09:41:41.6179689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6180035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6180335Z res = mod(**inputs) 2025-12-04T09:41:41.6180683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6181106Z outputs = self.model.decoder( 2025-12-04T09:41:41.6181472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6181857Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6182185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6182527Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6182876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6183232Z return func(*args, **kwargs) 2025-12-04T09:41:41.6183581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6183955Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6184089Z 2025-12-04T09:41:41.6184186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6184526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6184832Z res = mod(**inputs) 2025-12-04T09:41:41.6185162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6185528Z outputs = self.model.decoder( 2025-12-04T09:41:41.6185885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6186249Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6186573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6186912Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6187267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6187619Z return func(*args, **kwargs) 2025-12-04T09:41:41.6187970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6188361Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6188738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6189079Z return func(*args, **kwargs) 2025-12-04T09:41:41.6189434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.6189889Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.6190083Z 2025-12-04T09:41:41.6190187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6190521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6190827Z res = mod(**inputs) 2025-12-04T09:41:41.6191189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6191577Z outputs = self.model.decoder( 2025-12-04T09:41:41.6191933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6192292Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6192618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6192957Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6193314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6193665Z return func(*args, **kwargs) 2025-12-04T09:41:41.6194039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6194423Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6194801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6195150Z return func(*args, **kwargs) 2025-12-04T09:41:41.6195496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6195868Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6196000Z 2025-12-04T09:41:41.6196096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6196431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6196726Z res = mod(**inputs) 2025-12-04T09:41:41.6197071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6197443Z outputs = self.model.decoder( 2025-12-04T09:41:41.6197809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6198168Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6198501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6198846Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6199200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6199555Z return func(*args, **kwargs) 2025-12-04T09:41:41.6199908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6200301Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6200668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6201021Z return func(*args, **kwargs) 2025-12-04T09:41:41.6201375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6201749Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6201887Z 2025-12-04T09:41:41.6201965Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6202187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6202517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6202829Z res = mod(**inputs) 2025-12-04T09:41:41.6203172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6203549Z outputs = self.model.decoder( 2025-12-04T09:41:41.6203891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6204261Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6204601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6204934Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6205277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6205632Z return func(*args, **kwargs) 2025-12-04T09:41:41.6205985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6206375Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6206747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6207113Z return func(*args, **kwargs) 2025-12-04T09:41:41.6207466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6207847Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6208270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6208725Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6208730Z 2025-12-04T09:41:41.6208826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6209021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6209081Z res = mod(**inputs) 2025-12-04T09:41:41.6209324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6209403Z outputs = self.model.decoder( 2025-12-04T09:41:41.6209646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6209722Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6209930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6210005Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6210245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6210308Z return func(*args, **kwargs) 2025-12-04T09:41:41.6210555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6210647Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6210873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6210946Z return func(*args, **kwargs) 2025-12-04T09:41:41.6211212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6211369Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6211376Z 2025-12-04T09:41:41.6211501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6211711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6211786Z res = mod(**inputs) 2025-12-04T09:41:41.6212086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6212165Z outputs = self.model.decoder( 2025-12-04T09:41:41.6212443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6212521Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6212772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6212887Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6213118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6213191Z return func(*args, **kwargs) 2025-12-04T09:41:41.6213429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6213545Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6213548Z 2025-12-04T09:41:41.6213655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6213840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6213927Z res = mod(**inputs) 2025-12-04T09:41:41.6214172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6214241Z outputs = self.model.decoder( 2025-12-04T09:41:41.6214492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6214561Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6214769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6214849Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6215079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6215150Z return func(*args, **kwargs) 2025-12-04T09:41:41.6215389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6215502Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6215713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6215781Z return self.act(input) 2025-12-04T09:41:41.6215784Z 2025-12-04T09:41:41.6215886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6216073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6216131Z res = mod(**inputs) 2025-12-04T09:41:41.6216381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6216448Z outputs = self.model.decoder( 2025-12-04T09:41:41.6216690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6216764Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6216976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6217056Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6217288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6217350Z return func(*args, **kwargs) 2025-12-04T09:41:41.6217599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6217673Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6217697Z 2025-12-04T09:41:41.6217800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6217985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6218045Z res = mod(**inputs) 2025-12-04T09:41:41.6218291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6218375Z outputs = self.model.decoder( 2025-12-04T09:41:41.6218638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6218710Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6218916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6218994Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6219222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6219286Z return func(*args, **kwargs) 2025-12-04T09:41:41.6219530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:41:41.6219622Z hidden_states = residual + hidden_states 2025-12-04T09:41:41.6219625Z 2025-12-04T09:41:41.6219724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6219918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6219978Z res = mod(**inputs) 2025-12-04T09:41:41.6220227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6220294Z outputs = self.model.decoder( 2025-12-04T09:41:41.6220539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6220614Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6220825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6220906Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6221137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6221203Z return func(*args, **kwargs) 2025-12-04T09:41:41.6221453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6221545Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6221774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6221846Z return func(*args, **kwargs) 2025-12-04T09:41:41.6222086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.6222233Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.6222236Z 2025-12-04T09:41:41.6222332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6222519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6222586Z res = mod(**inputs) 2025-12-04T09:41:41.6222832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6222899Z outputs = self.model.decoder( 2025-12-04T09:41:41.6223149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6223214Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6223449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6223525Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6223750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6223823Z return func(*args, **kwargs) 2025-12-04T09:41:41.6224080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6224206Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6224431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6224496Z return func(*args, **kwargs) 2025-12-04T09:41:41.6224741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6224817Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6224820Z 2025-12-04T09:41:41.6224915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6225106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6225183Z res = mod(**inputs) 2025-12-04T09:41:41.6225441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6225508Z outputs = self.model.decoder( 2025-12-04T09:41:41.6225754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6225828Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6226040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6226120Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6226352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6226418Z return func(*args, **kwargs) 2025-12-04T09:41:41.6226666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6226758Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6226990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6227063Z return func(*args, **kwargs) 2025-12-04T09:41:41.6227306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6227393Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6227396Z 2025-12-04T09:41:41.6227471Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6227565Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6227762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6227822Z res = mod(**inputs) 2025-12-04T09:41:41.6228070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6228145Z outputs = self.model.decoder( 2025-12-04T09:41:41.6228390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6228466Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6228680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6228753Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6228989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6229068Z return func(*args, **kwargs) 2025-12-04T09:41:41.6229324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6229411Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6229631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6229712Z return func(*args, **kwargs) 2025-12-04T09:41:41.6229957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6230048Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6230325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6230448Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6230451Z 2025-12-04T09:41:41.6230553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6230735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6230792Z res = mod(**inputs) 2025-12-04T09:41:41.6231063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6231132Z outputs = self.model.decoder( 2025-12-04T09:41:41.6231376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6231439Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6231640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6231717Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6231940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6232001Z return func(*args, **kwargs) 2025-12-04T09:41:41.6232247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6232337Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6232566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6232628Z return func(*args, **kwargs) 2025-12-04T09:41:41.6232862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6232945Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6232948Z 2025-12-04T09:41:41.6233042Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6233232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6233292Z res = mod(**inputs) 2025-12-04T09:41:41.6233528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6233601Z outputs = self.model.decoder( 2025-12-04T09:41:41.6233835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6233900Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6234114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6234185Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6234413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6234475Z return func(*args, **kwargs) 2025-12-04T09:41:41.6234733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6234851Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6234854Z 2025-12-04T09:41:41.6234946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6235128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6235210Z res = mod(**inputs) 2025-12-04T09:41:41.6235463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6235538Z outputs = self.model.decoder( 2025-12-04T09:41:41.6235771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6235835Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6236048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6236120Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6236347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6236431Z return func(*args, **kwargs) 2025-12-04T09:41:41.6236670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6236789Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6236990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6237054Z return self.act(input) 2025-12-04T09:41:41.6237058Z 2025-12-04T09:41:41.6237160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6237341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6237407Z res = mod(**inputs) 2025-12-04T09:41:41.6237647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6237713Z outputs = self.model.decoder( 2025-12-04T09:41:41.6237955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6238020Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6238228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6238306Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6238530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6238597Z return func(*args, **kwargs) 2025-12-04T09:41:41.6238834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6238908Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6238911Z 2025-12-04T09:41:41.6239012Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6239197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6239262Z res = mod(**inputs) 2025-12-04T09:41:41.6239513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6239580Z outputs = self.model.decoder( 2025-12-04T09:41:41.6239833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6239900Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6240112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6240212Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6240452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6240521Z return func(*args, **kwargs) 2025-12-04T09:41:41.6240755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6240871Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6241123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6241187Z return func(*args, **kwargs) 2025-12-04T09:41:41.6241425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.6241573Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.6241577Z 2025-12-04T09:41:41.6241677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6241869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6241929Z res = mod(**inputs) 2025-12-04T09:41:41.6242193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6242268Z outputs = self.model.decoder( 2025-12-04T09:41:41.6242512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6242586Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6242797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6242868Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6243104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6243165Z return func(*args, **kwargs) 2025-12-04T09:41:41.6243403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6243503Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6243732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6243803Z return func(*args, **kwargs) 2025-12-04T09:41:41.6244039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6244112Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6244115Z 2025-12-04T09:41:41.6244220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6244405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6244472Z res = mod(**inputs) 2025-12-04T09:41:41.6244717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6244783Z outputs = self.model.decoder( 2025-12-04T09:41:41.6245030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6245096Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6245305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6245383Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6245608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6245678Z return func(*args, **kwargs) 2025-12-04T09:41:41.6245930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6246022Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6246256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6246321Z return func(*args, **kwargs) 2025-12-04T09:41:41.6246580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6246678Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6246682Z 2025-12-04T09:41:41.6246757Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6246861Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6247045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6247104Z res = mod(**inputs) 2025-12-04T09:41:41.6247357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6247426Z outputs = self.model.decoder( 2025-12-04T09:41:41.6247676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6247762Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6247978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6248063Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6248296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6248361Z return func(*args, **kwargs) 2025-12-04T09:41:41.6248612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6248706Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6248948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6249013Z return func(*args, **kwargs) 2025-12-04T09:41:41.6249259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6249361Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6249644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6249780Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6249783Z 2025-12-04T09:41:41.6249880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6250069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6250139Z res = mod(**inputs) 2025-12-04T09:41:41.6250386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6250455Z outputs = self.model.decoder( 2025-12-04T09:41:41.6250707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6250776Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6250998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6251073Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6251377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6251463Z return func(*args, **kwargs) 2025-12-04T09:41:41.6251760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6251874Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6252144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6252219Z return func(*args, **kwargs) 2025-12-04T09:41:41.6252518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6252627Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6252631Z 2025-12-04T09:41:41.6252744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6252973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6253036Z res = mod(**inputs) 2025-12-04T09:41:41.6253298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6253371Z outputs = self.model.decoder( 2025-12-04T09:41:41.6253623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6253701Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6253939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6254017Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6254265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6254330Z return func(*args, **kwargs) 2025-12-04T09:41:41.6254590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6254706Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6254710Z 2025-12-04T09:41:41.6254811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6255013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6255075Z res = mod(**inputs) 2025-12-04T09:41:41.6255334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6255407Z outputs = self.model.decoder( 2025-12-04T09:41:41.6255659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6255737Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6255957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6256032Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6256280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6256349Z return func(*args, **kwargs) 2025-12-04T09:41:41.6256609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6256726Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6256941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6257016Z return self.act(input) 2025-12-04T09:41:41.6257022Z 2025-12-04T09:41:41.6257122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6257324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6257384Z res = mod(**inputs) 2025-12-04T09:41:41.6257637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6257714Z outputs = self.model.decoder( 2025-12-04T09:41:41.6257985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6258057Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6258293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6258369Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6258636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6258719Z return func(*args, **kwargs) 2025-12-04T09:41:41.6258978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6259063Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6259066Z 2025-12-04T09:41:41.6259166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6259357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6259425Z res = mod(**inputs) 2025-12-04T09:41:41.6259668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6259760Z outputs = self.model.decoder( 2025-12-04T09:41:41.6260012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6260079Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6260304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6260377Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6260620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6260687Z return func(*args, **kwargs) 2025-12-04T09:41:41.6260938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:41:41.6261024Z hidden_states = residual + hidden_states 2025-12-04T09:41:41.6261028Z 2025-12-04T09:41:41.6261128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6261324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6261396Z res = mod(**inputs) 2025-12-04T09:41:41.6261652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6261731Z outputs = self.model.decoder( 2025-12-04T09:41:41.6261990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6262058Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6262293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6262370Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6262613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6262689Z return func(*args, **kwargs) 2025-12-04T09:41:41.6262961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6263065Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6263302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6263367Z return func(*args, **kwargs) 2025-12-04T09:41:41.6263624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.6264570Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.6264581Z 2025-12-04T09:41:41.6264699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6264896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6264965Z res = mod(**inputs) 2025-12-04T09:41:41.6265249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6265337Z outputs = self.model.decoder( 2025-12-04T09:41:41.6265588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6265665Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6265882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6265965Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6266203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6266269Z return func(*args, **kwargs) 2025-12-04T09:41:41.6266548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6266646Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6266895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6266959Z return func(*args, **kwargs) 2025-12-04T09:41:41.6267211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6267294Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6267298Z 2025-12-04T09:41:41.6267396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6267594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6267671Z res = mod(**inputs) 2025-12-04T09:41:41.6267924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6268003Z outputs = self.model.decoder( 2025-12-04T09:41:41.6268260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6268332Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6268564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6268641Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6268886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6268954Z return func(*args, **kwargs) 2025-12-04T09:41:41.6269206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6269308Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6269548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6269614Z return func(*args, **kwargs) 2025-12-04T09:41:41.6269876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6269960Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6269964Z 2025-12-04T09:41:41.6270051Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6270151Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6270347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6270460Z res = mod(**inputs) 2025-12-04T09:41:41.6270713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6270783Z outputs = self.model.decoder( 2025-12-04T09:41:41.6271046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6271133Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6271376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6271453Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6271690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6271763Z return func(*args, **kwargs) 2025-12-04T09:41:41.6272016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6272118Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6272528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6272639Z return func(*args, **kwargs) 2025-12-04T09:41:41.6272904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6273003Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6273297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6273441Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6273445Z 2025-12-04T09:41:41.6273547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6273754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6273818Z res = mod(**inputs) 2025-12-04T09:41:41.6274075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6274157Z outputs = self.model.decoder( 2025-12-04T09:41:41.6274418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6274497Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6274727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6274803Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6275061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6275125Z return func(*args, **kwargs) 2025-12-04T09:41:41.6275373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6275476Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6275712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6275782Z return func(*args, **kwargs) 2025-12-04T09:41:41.6276029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6276108Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6276112Z 2025-12-04T09:41:41.6276217Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6276406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6276466Z res = mod(**inputs) 2025-12-04T09:41:41.6276745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6276818Z outputs = self.model.decoder( 2025-12-04T09:41:41.6277070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6277142Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6277381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6277496Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6277727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6277799Z return func(*args, **kwargs) 2025-12-04T09:41:41.6278043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6278155Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6278159Z 2025-12-04T09:41:41.6278265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6278455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6278535Z res = mod(**inputs) 2025-12-04T09:41:41.6278789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6278860Z outputs = self.model.decoder( 2025-12-04T09:41:41.6279116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6279184Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6279399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6279480Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6279715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6279787Z return func(*args, **kwargs) 2025-12-04T09:41:41.6280031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6280145Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6280362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6280431Z return self.act(input) 2025-12-04T09:41:41.6280434Z 2025-12-04T09:41:41.6280532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6280728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6280787Z res = mod(**inputs) 2025-12-04T09:41:41.6281041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6281109Z outputs = self.model.decoder( 2025-12-04T09:41:41.6281353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6281430Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6281647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6281721Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6281961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6282024Z return func(*args, **kwargs) 2025-12-04T09:41:41.6282274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6282349Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6282372Z 2025-12-04T09:41:41.6282474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6282669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6282731Z res = mod(**inputs) 2025-12-04T09:41:41.6283167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6283274Z outputs = self.model.decoder( 2025-12-04T09:41:41.6283538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6283613Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6283827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6283901Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6284143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6284208Z return func(*args, **kwargs) 2025-12-04T09:41:41.6284460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6284585Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6284829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6284901Z return func(*args, **kwargs) 2025-12-04T09:41:41.6285138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T09:41:41.6285286Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:41:41.6285290Z 2025-12-04T09:41:41.6285387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6285574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6285642Z res = mod(**inputs) 2025-12-04T09:41:41.6285886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6285956Z outputs = self.model.decoder( 2025-12-04T09:41:41.6286210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6286279Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6286500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6286575Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6286804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6286875Z return func(*args, **kwargs) 2025-12-04T09:41:41.6287119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6287211Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6287448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6287514Z return func(*args, **kwargs) 2025-12-04T09:41:41.6287763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T09:41:41.6287841Z key_states = self.k_proj(current_states) 2025-12-04T09:41:41.6287844Z 2025-12-04T09:41:41.6287940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6288137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6288196Z res = mod(**inputs) 2025-12-04T09:41:41.6288461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6288532Z outputs = self.model.decoder( 2025-12-04T09:41:41.6288779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6288858Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6289093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6289185Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6289425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6289490Z return func(*args, **kwargs) 2025-12-04T09:41:41.6289741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6289834Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6290066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6290135Z return func(*args, **kwargs) 2025-12-04T09:41:41.6290396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T09:41:41.6290487Z value_states = self.v_proj(current_states) 2025-12-04T09:41:41.6290492Z 2025-12-04T09:41:41.6290569Z cudagraph partition due to non gpu ops 2025-12-04T09:41:41.6290665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6290859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6290919Z res = mod(**inputs) 2025-12-04T09:41:41.6291163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6291295Z outputs = self.model.decoder( 2025-12-04T09:41:41.6291556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6291630Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6291851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6291934Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6292206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6292277Z return func(*args, **kwargs) 2025-12-04T09:41:41.6292563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6292671Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6292930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6293009Z return func(*args, **kwargs) 2025-12-04T09:41:41.6293278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T09:41:41.6293381Z attn_output, attn_weights = attention_interface( 2025-12-04T09:41:41.6293708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:41:41.6293852Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:41:41.6293856Z 2025-12-04T09:41:41.6293980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6294179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6294241Z res = mod(**inputs) 2025-12-04T09:41:41.6294529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6294604Z outputs = self.model.decoder( 2025-12-04T09:41:41.6294858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6294938Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6295174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6295277Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6295522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6295591Z return func(*args, **kwargs) 2025-12-04T09:41:41.6295848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T09:41:41.6295944Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:41:41.6296188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6296256Z return func(*args, **kwargs) 2025-12-04T09:41:41.6296505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T09:41:41.6296614Z attn_output = self.out_proj(attn_output) 2025-12-04T09:41:41.6296619Z 2025-12-04T09:41:41.6296721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6296918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6296988Z res = mod(**inputs) 2025-12-04T09:41:41.6297240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6297317Z outputs = self.model.decoder( 2025-12-04T09:41:41.6297579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6297646Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6297870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6297946Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6298178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6298250Z return func(*args, **kwargs) 2025-12-04T09:41:41.6298492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6298611Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6298615Z 2025-12-04T09:41:41.6298711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6298900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6298969Z res = mod(**inputs) 2025-12-04T09:41:41.6299216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6299297Z outputs = self.model.decoder( 2025-12-04T09:41:41.6299539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6299606Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6299824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6299898Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6300131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6300202Z return func(*args, **kwargs) 2025-12-04T09:41:41.6300474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T09:41:41.6300593Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:41:41.6300800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:41:41.6300867Z return self.act(input) 2025-12-04T09:41:41.6300871Z 2025-12-04T09:41:41.6300990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6301194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6301260Z res = mod(**inputs) 2025-12-04T09:41:41.6301503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6301570Z outputs = self.model.decoder( 2025-12-04T09:41:41.6301817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6301883Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6302090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6302191Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6302420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6302493Z return func(*args, **kwargs) 2025-12-04T09:41:41.6302731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T09:41:41.6302807Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:41:41.6302810Z 2025-12-04T09:41:41.6302912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6303096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6303156Z res = mod(**inputs) 2025-12-04T09:41:41.6303401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T09:41:41.6303467Z outputs = self.model.decoder( 2025-12-04T09:41:41.6303713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T09:41:41.6303780Z layer_outputs = decoder_layer( 2025-12-04T09:41:41.6303992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:41:41.6304071Z return super().__call__(*args, **kwargs) 2025-12-04T09:41:41.6304296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:41:41.6304364Z return func(*args, **kwargs) 2025-12-04T09:41:41.6304603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T09:41:41.6304678Z hidden_states = residual + hidden_states 2025-12-04T09:41:41.6304681Z 2025-12-04T09:41:41.6304781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6304967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6305024Z res = mod(**inputs) 2025-12-04T09:41:41.6305273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1881, in forward 2025-12-04T09:41:41.6305346Z logits = self.lm_head(outputs[0]) 2025-12-04T09:41:41.6305350Z 2025-12-04T09:41:41.6305452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:41:41.6305635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:41:41.6305693Z res = mod(**inputs) 2025-12-04T09:41:41.6305958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1887, in forward 2025-12-04T09:41:41.6306097Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:41:41.6306101Z 2025-12-04T09:41:52.2451664Z Compilation time (from dynamo_timed): 16.455380563 2025-12-04T09:41:52.2795746Z pass 2025-12-04T09:41:52.2796339Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:41:52.2797542Z TIMING: _recursive_pre_grad_passes:0.00677 _recursive_joint_graph_passes:0.67209 _recursive_post_grad_passes:0.06629 async_compile.wait:0.85413 code_gen:9.79305 inductor_compile:11.07791 backend_compile:14.12439 gc:0.00035 entire_frame_compile:16.45538 total_wall_time:16.45538 2025-12-04T09:41:52.2798499Z STATS: call_* op count: 337 | FakeTensorMode.__torch_dispatch__:7389 | FakeTensor.__torch_dispatch__:4449 | ProxyTorchDispatchMode.__torch_dispatch__:2049 2025-12-04T09:41:52.2798973Z Dynamo produced 1 graphs covering 337 ops with 0 graph breaks (0 unique) 2025-12-04T09:41:54.7050056Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:41:54.7051195Z import pynvml # type: ignore[import] 2025-12-04T09:41:57.9938341Z 2025-12-04T09:42:00.0260387Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:42:00.0263723Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:42:00.0282801Z cpu eval MT5ForConditionalGeneration 2025-12-04T09:42:00.6484974Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:00.9103485Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:01.1744476Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:12.7329852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7330414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7330801Z res = mod(**inputs) 2025-12-04T09:42:12.7331246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7331879Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7332313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7332861Z layer_outputs = layer_module( 2025-12-04T09:42:12.7333262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7333625Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7334014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7334400Z return func(*args, **kwargs) 2025-12-04T09:42:12.7334773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7335166Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7335561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7335945Z return func(*args, **kwargs) 2025-12-04T09:42:12.7336308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7336714Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7337104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7337479Z return func(*args, **kwargs) 2025-12-04T09:42:12.7338211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 424, in forward 2025-12-04T09:42:12.7338625Z position_bias = position_bias + causal_mask 2025-12-04T09:42:12.7338770Z 2025-12-04T09:42:12.7338887Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7339253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7339633Z res = mod(**inputs) 2025-12-04T09:42:12.7340038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7340420Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7340788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7341202Z layer_outputs = layer_module( 2025-12-04T09:42:12.7341587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7341957Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7342334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7342760Z return func(*args, **kwargs) 2025-12-04T09:42:12.7343156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7343560Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7343944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7344319Z return func(*args, **kwargs) 2025-12-04T09:42:12.7344679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7359575Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7360203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7360615Z return func(*args, **kwargs) 2025-12-04T09:42:12.7360994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7361395Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7361548Z 2025-12-04T09:42:12.7361664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7362044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7362376Z res = mod(**inputs) 2025-12-04T09:42:12.7362751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7363155Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7363552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7363943Z layer_outputs = layer_module( 2025-12-04T09:42:12.7364304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7364681Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7365063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7365441Z return func(*args, **kwargs) 2025-12-04T09:42:12.7365832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7366250Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7366664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7367071Z return func(*args, **kwargs) 2025-12-04T09:42:12.7367545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7367987Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7368413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7368816Z return func(*args, **kwargs) 2025-12-04T09:42:12.7369240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7369679Z key_states = self.k(current_states) 2025-12-04T09:42:12.7369828Z 2025-12-04T09:42:12.7369944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7370335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7370679Z res = mod(**inputs) 2025-12-04T09:42:12.7371059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7371575Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7371983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7372731Z layer_outputs = layer_module( 2025-12-04T09:42:12.7373126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7373534Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7373953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7374354Z return func(*args, **kwargs) 2025-12-04T09:42:12.7374749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7375180Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7375608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7375998Z return func(*args, **kwargs) 2025-12-04T09:42:12.7376383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7376794Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7377198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7377601Z return func(*args, **kwargs) 2025-12-04T09:42:12.7377986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7378386Z value_states = self.v(current_states) 2025-12-04T09:42:12.7378530Z 2025-12-04T09:42:12.7378619Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7378851Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7379113Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7379493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7379839Z res = mod(**inputs) 2025-12-04T09:42:12.7380206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7380593Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7380961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7381339Z layer_outputs = layer_module( 2025-12-04T09:42:12.7381696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7382059Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7382456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7382929Z return func(*args, **kwargs) 2025-12-04T09:42:12.7383323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7383726Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7384130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7384556Z return func(*args, **kwargs) 2025-12-04T09:42:12.7384983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7385372Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7385746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7386114Z return func(*args, **kwargs) 2025-12-04T09:42:12.7386473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7386847Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7386991Z 2025-12-04T09:42:12.7387103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7387504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7387820Z res = mod(**inputs) 2025-12-04T09:42:12.7388161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7388533Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7388907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7389276Z layer_outputs = layer_module( 2025-12-04T09:42:12.7389629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7390001Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7390383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7390752Z return func(*args, **kwargs) 2025-12-04T09:42:12.7391115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7391529Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7391963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7392408Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7392849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7393691Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7393855Z 2025-12-04T09:42:12.7393968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7394351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7394697Z res = mod(**inputs) 2025-12-04T09:42:12.7395082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7395455Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7395843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7396245Z layer_outputs = layer_module( 2025-12-04T09:42:12.7396616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7397004Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7397438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7397842Z return func(*args, **kwargs) 2025-12-04T09:42:12.7398226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7398656Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7399093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7399556Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7399985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7400397Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7400545Z 2025-12-04T09:42:12.7400663Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7401052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7401404Z res = mod(**inputs) 2025-12-04T09:42:12.7401783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7402213Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7402601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7403003Z layer_outputs = layer_module( 2025-12-04T09:42:12.7403369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7403749Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7404139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7404531Z return func(*args, **kwargs) 2025-12-04T09:42:12.7404911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7405316Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7405728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7406173Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7406617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7407035Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7407192Z 2025-12-04T09:42:12.7407305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7407688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7408100Z res = mod(**inputs) 2025-12-04T09:42:12.7408472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7408881Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7409292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7409683Z layer_outputs = layer_module( 2025-12-04T09:42:12.7410057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7410447Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7410849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7411243Z return func(*args, **kwargs) 2025-12-04T09:42:12.7411715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7412158Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7412631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7413100Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7413561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7413994Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7414155Z 2025-12-04T09:42:12.7414288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7414684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7415048Z res = mod(**inputs) 2025-12-04T09:42:12.7415432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7415833Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7416257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7416673Z layer_outputs = layer_module( 2025-12-04T09:42:12.7417040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7417453Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7417866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7418268Z return func(*args, **kwargs) 2025-12-04T09:42:12.7418651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7419062Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7419472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7419866Z return func(*args, **kwargs) 2025-12-04T09:42:12.7420260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7420677Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7421174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7421627Z return func(*args, **kwargs) 2025-12-04T09:42:12.7422022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7422435Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7422581Z 2025-12-04T09:42:12.7422700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7423085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7423438Z res = mod(**inputs) 2025-12-04T09:42:12.7423822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7424224Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7424627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7425034Z layer_outputs = layer_module( 2025-12-04T09:42:12.7425415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7425803Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7426217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7426620Z return func(*args, **kwargs) 2025-12-04T09:42:12.7426991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7427389Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7427809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7428207Z return func(*args, **kwargs) 2025-12-04T09:42:12.7428589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7429008Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7429438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7429848Z return func(*args, **kwargs) 2025-12-04T09:42:12.7430236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7430637Z key_states = self.k(current_states) 2025-12-04T09:42:12.7430774Z 2025-12-04T09:42:12.7430891Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7431263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7431598Z res = mod(**inputs) 2025-12-04T09:42:12.7431964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7432380Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7432772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7433181Z layer_outputs = layer_module( 2025-12-04T09:42:12.7433547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7433930Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7434339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7434739Z return func(*args, **kwargs) 2025-12-04T09:42:12.7435118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7435514Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7435975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7436377Z return func(*args, **kwargs) 2025-12-04T09:42:12.7436760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7437230Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7437610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7437968Z return func(*args, **kwargs) 2025-12-04T09:42:12.7438326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7438711Z value_states = self.v(current_states) 2025-12-04T09:42:12.7438844Z 2025-12-04T09:42:12.7438933Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7439149Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7439400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7439778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7440109Z res = mod(**inputs) 2025-12-04T09:42:12.7440480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7440875Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7441268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7441658Z layer_outputs = layer_module( 2025-12-04T09:42:12.7442038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7442421Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7442816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7443216Z return func(*args, **kwargs) 2025-12-04T09:42:12.7443615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7444033Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7444426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7444810Z return func(*args, **kwargs) 2025-12-04T09:42:12.7445188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7445593Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7445998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7446384Z return func(*args, **kwargs) 2025-12-04T09:42:12.7446758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7447170Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7447312Z 2025-12-04T09:42:12.7447422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7447807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7448144Z res = mod(**inputs) 2025-12-04T09:42:12.7448505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7448901Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7449298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7449686Z layer_outputs = layer_module( 2025-12-04T09:42:12.7450056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7450450Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7450850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7451237Z return func(*args, **kwargs) 2025-12-04T09:42:12.7451701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7452128Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7452546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7453006Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7453426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7453867Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7454036Z 2025-12-04T09:42:12.7454147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7454542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7454897Z res = mod(**inputs) 2025-12-04T09:42:12.7455268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7455682Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7456087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7456489Z layer_outputs = layer_module( 2025-12-04T09:42:12.7456878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7457284Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7457706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7458131Z return func(*args, **kwargs) 2025-12-04T09:42:12.7458545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7458990Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7459412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7459855Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7460307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7460721Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7460867Z 2025-12-04T09:42:12.7460986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7461367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7461756Z res = mod(**inputs) 2025-12-04T09:42:12.7462136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7462548Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7462949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7463357Z layer_outputs = layer_module( 2025-12-04T09:42:12.7463735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7464126Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7464545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7464951Z return func(*args, **kwargs) 2025-12-04T09:42:12.7465347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7465772Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7466201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7466660Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7467105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7467527Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7467701Z 2025-12-04T09:42:12.7467811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7468194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7468533Z res = mod(**inputs) 2025-12-04T09:42:12.7468903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7469305Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7469701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7470092Z layer_outputs = layer_module( 2025-12-04T09:42:12.7470462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7470849Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7471244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7471636Z return func(*args, **kwargs) 2025-12-04T09:42:12.7472037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7472633Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7473047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7473540Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7474013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7474409Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7474560Z 2025-12-04T09:42:12.7474668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7475048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7475389Z res = mod(**inputs) 2025-12-04T09:42:12.7475757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7476158Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7476551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7476975Z layer_outputs = layer_module( 2025-12-04T09:42:12.7477335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7477719Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7478118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7478517Z return func(*args, **kwargs) 2025-12-04T09:42:12.7478890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7479295Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7479697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7480098Z return func(*args, **kwargs) 2025-12-04T09:42:12.7480478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7480903Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7481308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7481667Z return func(*args, **kwargs) 2025-12-04T09:42:12.7482027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7482407Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7482538Z 2025-12-04T09:42:12.7482650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7483001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7483322Z res = mod(**inputs) 2025-12-04T09:42:12.7483666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7484038Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7484410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7484784Z layer_outputs = layer_module( 2025-12-04T09:42:12.7485136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7485508Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7485904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7486331Z return func(*args, **kwargs) 2025-12-04T09:42:12.7486705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7487116Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7487516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7487930Z return func(*args, **kwargs) 2025-12-04T09:42:12.7488319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7488734Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7489145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7489554Z return func(*args, **kwargs) 2025-12-04T09:42:12.7489933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7490335Z key_states = self.k(current_states) 2025-12-04T09:42:12.7490473Z 2025-12-04T09:42:12.7490589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7490966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7491401Z res = mod(**inputs) 2025-12-04T09:42:12.7491785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7492198Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7492605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7493054Z layer_outputs = layer_module( 2025-12-04T09:42:12.7493427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7493814Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7494226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7494630Z return func(*args, **kwargs) 2025-12-04T09:42:12.7495015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7495429Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7495846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7496250Z return func(*args, **kwargs) 2025-12-04T09:42:12.7496631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7497049Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7497460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7497863Z return func(*args, **kwargs) 2025-12-04T09:42:12.7498244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7498696Z value_states = self.v(current_states) 2025-12-04T09:42:12.7498835Z 2025-12-04T09:42:12.7498925Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7499150Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7499397Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7499777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7500119Z res = mod(**inputs) 2025-12-04T09:42:12.7500485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7500896Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7501317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7501720Z layer_outputs = layer_module( 2025-12-04T09:42:12.7502081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7502468Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7502881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7503285Z return func(*args, **kwargs) 2025-12-04T09:42:12.7503664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7504062Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7504463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7504850Z return func(*args, **kwargs) 2025-12-04T09:42:12.7505231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7505632Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7506050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7506448Z return func(*args, **kwargs) 2025-12-04T09:42:12.7506826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7507229Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7507367Z 2025-12-04T09:42:12.7507478Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7507859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7508200Z res = mod(**inputs) 2025-12-04T09:42:12.7508565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7508967Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7509360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7509760Z layer_outputs = layer_module( 2025-12-04T09:42:12.7510120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7510508Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7510912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7511310Z return func(*args, **kwargs) 2025-12-04T09:42:12.7511685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7512105Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7512525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7512964Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7513404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7513811Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7513972Z 2025-12-04T09:42:12.7514091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7514460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7514801Z res = mod(**inputs) 2025-12-04T09:42:12.7515171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7515571Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7515971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7516352Z layer_outputs = layer_module( 2025-12-04T09:42:12.7516720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7517102Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7517519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7517940Z return func(*args, **kwargs) 2025-12-04T09:42:12.7518317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7518720Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7519134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7519570Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7519995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7520446Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7520591Z 2025-12-04T09:42:12.7520700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7521076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7521407Z res = mod(**inputs) 2025-12-04T09:42:12.7521773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7522178Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7522567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7522959Z layer_outputs = layer_module( 2025-12-04T09:42:12.7523305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7523677Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7524064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7524452Z return func(*args, **kwargs) 2025-12-04T09:42:12.7524834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7525245Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7525646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7526086Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7526519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7526925Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7527081Z 2025-12-04T09:42:12.7527188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7527566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7527908Z res = mod(**inputs) 2025-12-04T09:42:12.7528268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7528669Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7529062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7529466Z layer_outputs = layer_module( 2025-12-04T09:42:12.7529824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7530231Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7530640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7531043Z return func(*args, **kwargs) 2025-12-04T09:42:12.7531539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7531997Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7532447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7532893Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7533345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7533766Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7533908Z 2025-12-04T09:42:12.7534022Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7534424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7534765Z res = mod(**inputs) 2025-12-04T09:42:12.7535158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7535552Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7535963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7536379Z layer_outputs = layer_module( 2025-12-04T09:42:12.7536754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7537143Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7537560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7537968Z return func(*args, **kwargs) 2025-12-04T09:42:12.7538347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7538769Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7539190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7539597Z return func(*args, **kwargs) 2025-12-04T09:42:12.7539986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7540402Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7540813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7541199Z return func(*args, **kwargs) 2025-12-04T09:42:12.7541586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7541988Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7542131Z 2025-12-04T09:42:12.7542249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7542628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7542981Z res = mod(**inputs) 2025-12-04T09:42:12.7543340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7543718Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7544096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7544476Z layer_outputs = layer_module( 2025-12-04T09:42:12.7544829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7545212Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7545596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7545968Z return func(*args, **kwargs) 2025-12-04T09:42:12.7546325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7546720Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7547121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7547492Z return func(*args, **kwargs) 2025-12-04T09:42:12.7547842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7548230Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7548616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7548984Z return func(*args, **kwargs) 2025-12-04T09:42:12.7549336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7549736Z key_states = self.k(current_states) 2025-12-04T09:42:12.7549867Z 2025-12-04T09:42:12.7549977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7550321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7550647Z res = mod(**inputs) 2025-12-04T09:42:12.7550995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7551373Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7551737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7552115Z layer_outputs = layer_module( 2025-12-04T09:42:12.7552465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7552828Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7553202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7553574Z return func(*args, **kwargs) 2025-12-04T09:42:12.7553933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7554309Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7554695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7555065Z return func(*args, **kwargs) 2025-12-04T09:42:12.7555427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7555807Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7556187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7556558Z return func(*args, **kwargs) 2025-12-04T09:42:12.7556910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7557295Z value_states = self.v(current_states) 2025-12-04T09:42:12.7557434Z 2025-12-04T09:42:12.7557515Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7557726Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7557953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7558316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7558631Z res = mod(**inputs) 2025-12-04T09:42:12.7558994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7559361Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7559723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7560091Z layer_outputs = layer_module( 2025-12-04T09:42:12.7560439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7560809Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7561179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7561535Z return func(*args, **kwargs) 2025-12-04T09:42:12.7561892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7562264Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7562636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7562989Z return func(*args, **kwargs) 2025-12-04T09:42:12.7563362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7563739Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7564112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7564469Z return func(*args, **kwargs) 2025-12-04T09:42:12.7564822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7565200Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7565326Z 2025-12-04T09:42:12.7565428Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7565780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7566095Z res = mod(**inputs) 2025-12-04T09:42:12.7566436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7566799Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7567164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7567528Z layer_outputs = layer_module( 2025-12-04T09:42:12.7567859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7568217Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7568582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7568950Z return func(*args, **kwargs) 2025-12-04T09:42:12.7569296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7569667Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7570043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7570404Z return func(*args, **kwargs) 2025-12-04T09:42:12.7570751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:42:12.7571178Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:42:12.7571436Z 2025-12-04T09:42:12.7571548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7571894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7572248Z res = mod(**inputs) 2025-12-04T09:42:12.7572818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7573243Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7573636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7574006Z layer_outputs = layer_module( 2025-12-04T09:42:12.7574389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7574771Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7575149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7575516Z return func(*args, **kwargs) 2025-12-04T09:42:12.7575872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7576257Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7576643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7577097Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7577507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7577897Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7578057Z 2025-12-04T09:42:12.7578158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7578511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7578818Z res = mod(**inputs) 2025-12-04T09:42:12.7579164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7579540Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7579904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7580271Z layer_outputs = layer_module( 2025-12-04T09:42:12.7580618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7580978Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7581348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7581715Z return func(*args, **kwargs) 2025-12-04T09:42:12.7582073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7582462Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7582841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7583254Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7583663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7584040Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7584171Z 2025-12-04T09:42:12.7584274Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7584632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7584950Z res = mod(**inputs) 2025-12-04T09:42:12.7585284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7585657Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7586039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7586416Z layer_outputs = layer_module( 2025-12-04T09:42:12.7586750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7587111Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7587485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7587863Z return func(*args, **kwargs) 2025-12-04T09:42:12.7588243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7588631Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7589017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7589425Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7589834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7590218Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7590359Z 2025-12-04T09:42:12.7590487Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7590838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7591158Z res = mod(**inputs) 2025-12-04T09:42:12.7591504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7591870Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7592234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7592611Z layer_outputs = layer_module( 2025-12-04T09:42:12.7592959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7593310Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7593682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7594053Z return func(*args, **kwargs) 2025-12-04T09:42:12.7594403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7594796Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7595182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7595596Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7595995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7596373Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7596506Z 2025-12-04T09:42:12.7596617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7596970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7597285Z res = mod(**inputs) 2025-12-04T09:42:12.7597636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7598011Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7598376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7598753Z layer_outputs = layer_module( 2025-12-04T09:42:12.7599097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7599453Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7599854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7600226Z return func(*args, **kwargs) 2025-12-04T09:42:12.7600583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7600961Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7601364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7601755Z return func(*args, **kwargs) 2025-12-04T09:42:12.7602122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7602510Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7602919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7603315Z return func(*args, **kwargs) 2025-12-04T09:42:12.7603700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7604105Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7604275Z 2025-12-04T09:42:12.7604386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7604771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7605108Z res = mod(**inputs) 2025-12-04T09:42:12.7605482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7605880Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7606276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7606674Z layer_outputs = layer_module( 2025-12-04T09:42:12.7607047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7607433Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7607828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7608227Z return func(*args, **kwargs) 2025-12-04T09:42:12.7608609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7609017Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7609413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7609806Z return func(*args, **kwargs) 2025-12-04T09:42:12.7610186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7610600Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7610997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7611466Z return func(*args, **kwargs) 2025-12-04T09:42:12.7611856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7612267Z key_states = self.k(current_states) 2025-12-04T09:42:12.7612419Z 2025-12-04T09:42:12.7612534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7612925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7613291Z res = mod(**inputs) 2025-12-04T09:42:12.7613710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7614112Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7614530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7614939Z layer_outputs = layer_module( 2025-12-04T09:42:12.7615306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7615709Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7616134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7616547Z return func(*args, **kwargs) 2025-12-04T09:42:12.7616930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7617345Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7617746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7618151Z return func(*args, **kwargs) 2025-12-04T09:42:12.7618537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7618954Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7619361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7619788Z return func(*args, **kwargs) 2025-12-04T09:42:12.7620184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7620603Z value_states = self.v(current_states) 2025-12-04T09:42:12.7620744Z 2025-12-04T09:42:12.7620828Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7621058Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7621308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7621692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7622053Z res = mod(**inputs) 2025-12-04T09:42:12.7622422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7622812Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7623185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7623569Z layer_outputs = layer_module( 2025-12-04T09:42:12.7623920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7624282Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7624666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7625047Z return func(*args, **kwargs) 2025-12-04T09:42:12.7625410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7625791Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7626180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7626562Z return func(*args, **kwargs) 2025-12-04T09:42:12.7626920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7627308Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7627694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7628066Z return func(*args, **kwargs) 2025-12-04T09:42:12.7628422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7628806Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7628935Z 2025-12-04T09:42:12.7629080Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7629445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7629757Z res = mod(**inputs) 2025-12-04T09:42:12.7630104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7630496Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7630887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7631264Z layer_outputs = layer_module( 2025-12-04T09:42:12.7631611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7631975Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7632347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7632724Z return func(*args, **kwargs) 2025-12-04T09:42:12.7633106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7633537Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7633949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7634392Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7634823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7635242Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7635402Z 2025-12-04T09:42:12.7635506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7635864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7636181Z res = mod(**inputs) 2025-12-04T09:42:12.7636526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7636900Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7637272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7637643Z layer_outputs = layer_module( 2025-12-04T09:42:12.7638005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7638384Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7638775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7639153Z return func(*args, **kwargs) 2025-12-04T09:42:12.7639540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7639956Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7640356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7640795Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7641221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7641612Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7641755Z 2025-12-04T09:42:12.7641865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7642236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7642574Z res = mod(**inputs) 2025-12-04T09:42:12.7642960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7643357Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7643748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7644140Z layer_outputs = layer_module( 2025-12-04T09:42:12.7644518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7644928Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7645339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7645749Z return func(*args, **kwargs) 2025-12-04T09:42:12.7646131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7646555Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7646975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7647416Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7647869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7648288Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7648436Z 2025-12-04T09:42:12.7648552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7648919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7649257Z res = mod(**inputs) 2025-12-04T09:42:12.7649624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7650019Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7650402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7650796Z layer_outputs = layer_module( 2025-12-04T09:42:12.7651159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7651641Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7652065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7652475Z return func(*args, **kwargs) 2025-12-04T09:42:12.7652871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7653280Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7653694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7654149Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7654580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7654998Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7655152Z 2025-12-04T09:42:12.7655262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7655658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7656010Z res = mod(**inputs) 2025-12-04T09:42:12.7656378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7656774Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7657170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7657627Z layer_outputs = layer_module( 2025-12-04T09:42:12.7658018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7658415Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7658806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7659209Z return func(*args, **kwargs) 2025-12-04T09:42:12.7659604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7660038Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7660451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7660851Z return func(*args, **kwargs) 2025-12-04T09:42:12.7661236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7661650Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7662051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7662424Z return func(*args, **kwargs) 2025-12-04T09:42:12.7662810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7663190Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7663335Z 2025-12-04T09:42:12.7663441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7663810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7664127Z res = mod(**inputs) 2025-12-04T09:42:12.7664478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7664854Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7665226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7665604Z layer_outputs = layer_module( 2025-12-04T09:42:12.7665949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7666318Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7666701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7667068Z return func(*args, **kwargs) 2025-12-04T09:42:12.7667428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7667812Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7668187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7668561Z return func(*args, **kwargs) 2025-12-04T09:42:12.7668924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7669313Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7669703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7670095Z return func(*args, **kwargs) 2025-12-04T09:42:12.7670474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7670851Z key_states = self.k(current_states) 2025-12-04T09:42:12.7670992Z 2025-12-04T09:42:12.7671096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7671460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7671784Z res = mod(**inputs) 2025-12-04T09:42:12.7672149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7672683Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7673062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7673447Z layer_outputs = layer_module( 2025-12-04T09:42:12.7673838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7674226Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7674601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7674964Z return func(*args, **kwargs) 2025-12-04T09:42:12.7675324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7675704Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7676084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7676445Z return func(*args, **kwargs) 2025-12-04T09:42:12.7676853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7677242Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7677619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7677990Z return func(*args, **kwargs) 2025-12-04T09:42:12.7678348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7678726Z value_states = self.v(current_states) 2025-12-04T09:42:12.7678859Z 2025-12-04T09:42:12.7678938Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7679152Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7679390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7679742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7680072Z res = mod(**inputs) 2025-12-04T09:42:12.7680421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7680797Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7681155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7681526Z layer_outputs = layer_module( 2025-12-04T09:42:12.7681871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7682224Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7682601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7682969Z return func(*args, **kwargs) 2025-12-04T09:42:12.7683325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7683699Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7684082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7684452Z return func(*args, **kwargs) 2025-12-04T09:42:12.7684811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7685188Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7685574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7685970Z return func(*args, **kwargs) 2025-12-04T09:42:12.7686328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7686716Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7686863Z 2025-12-04T09:42:12.7686976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7687426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7687783Z res = mod(**inputs) 2025-12-04T09:42:12.7688167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7688570Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7688971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7689386Z layer_outputs = layer_module( 2025-12-04T09:42:12.7689770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7690175Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7690586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7690998Z return func(*args, **kwargs) 2025-12-04T09:42:12.7691458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7691869Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7692273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7692681Z return func(*args, **kwargs) 2025-12-04T09:42:12.7693059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:42:12.7693534Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:42:12.7693743Z 2025-12-04T09:42:12.7693854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7694232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7694575Z res = mod(**inputs) 2025-12-04T09:42:12.7694942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7695339Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7695731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7696142Z layer_outputs = layer_module( 2025-12-04T09:42:12.7696510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7696918Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7697321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7697714Z return func(*args, **kwargs) 2025-12-04T09:42:12.7698093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7698520Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7698938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7699387Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7699833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7700264Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7700424Z 2025-12-04T09:42:12.7700533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7700947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7701293Z res = mod(**inputs) 2025-12-04T09:42:12.7701642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7702009Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7702398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7702789Z layer_outputs = layer_module( 2025-12-04T09:42:12.7703126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7703492Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7703868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7704241Z return func(*args, **kwargs) 2025-12-04T09:42:12.7704593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7704983Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7705397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7705817Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7706221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7706602Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7706733Z 2025-12-04T09:42:12.7706842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7707188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7707510Z res = mod(**inputs) 2025-12-04T09:42:12.7707856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7708228Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7708589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7708961Z layer_outputs = layer_module( 2025-12-04T09:42:12.7709307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7709660Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7710038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7710409Z return func(*args, **kwargs) 2025-12-04T09:42:12.7710767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7711155Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7711540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7711954Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7712366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7712748Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7712899Z 2025-12-04T09:42:12.7713004Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7713359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7713668Z res = mod(**inputs) 2025-12-04T09:42:12.7714011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7714387Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7714774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7715142Z layer_outputs = layer_module( 2025-12-04T09:42:12.7715488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7715852Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7716239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7716618Z return func(*args, **kwargs) 2025-12-04T09:42:12.7716975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7717365Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7717751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7718165Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7718578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7718975Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7719106Z 2025-12-04T09:42:12.7719208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7719558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7719876Z res = mod(**inputs) 2025-12-04T09:42:12.7720205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7720571Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7720932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7721293Z layer_outputs = layer_module( 2025-12-04T09:42:12.7721626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7721990Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7722374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7722742Z return func(*args, **kwargs) 2025-12-04T09:42:12.7723104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7723486Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7723868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7724232Z return func(*args, **kwargs) 2025-12-04T09:42:12.7724595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7724980Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7725362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7725721Z return func(*args, **kwargs) 2025-12-04T09:42:12.7726080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7726463Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7726604Z 2025-12-04T09:42:12.7726712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7727088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7727425Z res = mod(**inputs) 2025-12-04T09:42:12.7727797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7728208Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7728600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7728991Z layer_outputs = layer_module( 2025-12-04T09:42:12.7729349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7729748Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7730177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7730573Z return func(*args, **kwargs) 2025-12-04T09:42:12.7730956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7731443Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7731857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7732244Z return func(*args, **kwargs) 2025-12-04T09:42:12.7732627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7733064Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7733479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7733867Z return func(*args, **kwargs) 2025-12-04T09:42:12.7734248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7734647Z key_states = self.k(current_states) 2025-12-04T09:42:12.7734785Z 2025-12-04T09:42:12.7734900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7735269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7735606Z res = mod(**inputs) 2025-12-04T09:42:12.7735972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7736049Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7736308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7736390Z layer_outputs = layer_module( 2025-12-04T09:42:12.7736627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7736716Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7736969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7737040Z return func(*args, **kwargs) 2025-12-04T09:42:12.7737304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7737387Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7737640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7737719Z return func(*args, **kwargs) 2025-12-04T09:42:12.7737970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7738066Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7738319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7738388Z return func(*args, **kwargs) 2025-12-04T09:42:12.7738649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7738731Z value_states = self.v(current_states) 2025-12-04T09:42:12.7738735Z 2025-12-04T09:42:12.7738850Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7738936Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7739047Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7739265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7739334Z res = mod(**inputs) 2025-12-04T09:42:12.7739612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7740517Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7740772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7740853Z layer_outputs = layer_module( 2025-12-04T09:42:12.7741078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7741159Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7741414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7741483Z return func(*args, **kwargs) 2025-12-04T09:42:12.7741744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7741833Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7742082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7742158Z return func(*args, **kwargs) 2025-12-04T09:42:12.7742403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7742483Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7742739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7742803Z return func(*args, **kwargs) 2025-12-04T09:42:12.7743055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7743133Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7743137Z 2025-12-04T09:42:12.7743242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7743451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7743514Z res = mod(**inputs) 2025-12-04T09:42:12.7743762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7743843Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7744089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7744166Z layer_outputs = layer_module( 2025-12-04T09:42:12.7744395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7744473Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7744726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7744791Z return func(*args, **kwargs) 2025-12-04T09:42:12.7745036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7745137Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7745380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7745505Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7745771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7745872Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7745875Z 2025-12-04T09:42:12.7745986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7746184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7746255Z res = mod(**inputs) 2025-12-04T09:42:12.7746514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7746605Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7746852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7746923Z layer_outputs = layer_module( 2025-12-04T09:42:12.7747141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7747227Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7747468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7747542Z return func(*args, **kwargs) 2025-12-04T09:42:12.7747798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7747888Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7748144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7748257Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7748501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7748580Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7748583Z 2025-12-04T09:42:12.7748686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7748890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7748951Z res = mod(**inputs) 2025-12-04T09:42:12.7749193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7749274Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7749515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7749594Z layer_outputs = layer_module( 2025-12-04T09:42:12.7749813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7749887Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7750135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7750204Z return func(*args, **kwargs) 2025-12-04T09:42:12.7750441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7750538Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7750777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7750909Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7751138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7751222Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7751226Z 2025-12-04T09:42:12.7751333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7751528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7751617Z res = mod(**inputs) 2025-12-04T09:42:12.7751860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7751929Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7752177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7752272Z layer_outputs = layer_module( 2025-12-04T09:42:12.7752496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7752574Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7752800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7752870Z return func(*args, **kwargs) 2025-12-04T09:42:12.7753097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7753180Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7753415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7753541Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7753782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7753858Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7753862Z 2025-12-04T09:42:12.7753958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7754150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7754210Z res = mod(**inputs) 2025-12-04T09:42:12.7754443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7754520Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7754747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7754821Z layer_outputs = layer_module( 2025-12-04T09:42:12.7755032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7755107Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7755345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7755410Z return func(*args, **kwargs) 2025-12-04T09:42:12.7755636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7755717Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7755947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7756022Z return func(*args, **kwargs) 2025-12-04T09:42:12.7756255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7756334Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7756574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7756649Z return func(*args, **kwargs) 2025-12-04T09:42:12.7756886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7756958Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7756962Z 2025-12-04T09:42:12.7757060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7757269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7757329Z res = mod(**inputs) 2025-12-04T09:42:12.7757559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7757636Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7757892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7757967Z layer_outputs = layer_module( 2025-12-04T09:42:12.7758193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7758266Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7758502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7758567Z return func(*args, **kwargs) 2025-12-04T09:42:12.7758795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7758878Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7759101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7759188Z return func(*args, **kwargs) 2025-12-04T09:42:12.7759417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7759496Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7759734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7759797Z return func(*args, **kwargs) 2025-12-04T09:42:12.7760030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7760103Z key_states = self.k(current_states) 2025-12-04T09:42:12.7760108Z 2025-12-04T09:42:12.7760205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7760396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7760457Z res = mod(**inputs) 2025-12-04T09:42:12.7760688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7760767Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7760998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7761071Z layer_outputs = layer_module( 2025-12-04T09:42:12.7761277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7761349Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7761584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7761647Z return func(*args, **kwargs) 2025-12-04T09:42:12.7761873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7761957Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7762183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7762255Z return func(*args, **kwargs) 2025-12-04T09:42:12.7762484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7762560Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7762796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7762861Z return func(*args, **kwargs) 2025-12-04T09:42:12.7763111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7763185Z value_states = self.v(current_states) 2025-12-04T09:42:12.7763188Z 2025-12-04T09:42:12.7763266Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7763345Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7763462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7763662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7763729Z res = mod(**inputs) 2025-12-04T09:42:12.7763966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7764041Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7764273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7764341Z layer_outputs = layer_module( 2025-12-04T09:42:12.7764556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7764626Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7764877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7764948Z return func(*args, **kwargs) 2025-12-04T09:42:12.7765178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7765260Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7765491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7765555Z return func(*args, **kwargs) 2025-12-04T09:42:12.7765793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7765870Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7766109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7766173Z return func(*args, **kwargs) 2025-12-04T09:42:12.7766404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7766488Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7766492Z 2025-12-04T09:42:12.7766590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7766779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7766848Z res = mod(**inputs) 2025-12-04T09:42:12.7767082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7767161Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7767397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7767463Z layer_outputs = layer_module( 2025-12-04T09:42:12.7767686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7767761Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7767997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7768069Z return func(*args, **kwargs) 2025-12-04T09:42:12.7768300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7768383Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7768634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7768702Z return func(*args, **kwargs) 2025-12-04T09:42:12.7768940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7769020Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7769277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7769359Z return func(*args, **kwargs) 2025-12-04T09:42:12.7769593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7769672Z key_states = self.k(current_states) 2025-12-04T09:42:12.7769676Z 2025-12-04T09:42:12.7769775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7769970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7770043Z res = mod(**inputs) 2025-12-04T09:42:12.7770282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7770362Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7770625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7770695Z layer_outputs = layer_module( 2025-12-04T09:42:12.7770924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7771001Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7771240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7771401Z return func(*args, **kwargs) 2025-12-04T09:42:12.7771653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7771752Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7772013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7772089Z return func(*args, **kwargs) 2025-12-04T09:42:12.7772484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7772578Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7772845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7772915Z return func(*args, **kwargs) 2025-12-04T09:42:12.7773169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7773254Z value_states = self.v(current_states) 2025-12-04T09:42:12.7773258Z 2025-12-04T09:42:12.7773339Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7773416Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7773527Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7773720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7773793Z res = mod(**inputs) 2025-12-04T09:42:12.7774034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7774107Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7774356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7774426Z layer_outputs = layer_module( 2025-12-04T09:42:12.7774646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7774775Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7775016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7775090Z return func(*args, **kwargs) 2025-12-04T09:42:12.7775331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7775432Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7775705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7775771Z return func(*args, **kwargs) 2025-12-04T09:42:12.7776012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7776101Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7776342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7776417Z return func(*args, **kwargs) 2025-12-04T09:42:12.7776654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7776761Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7776765Z 2025-12-04T09:42:12.7776876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7777072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7777142Z res = mod(**inputs) 2025-12-04T09:42:12.7777381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7777452Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7777699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7777773Z layer_outputs = layer_module( 2025-12-04T09:42:12.7777993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7778080Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7778323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7778398Z return func(*args, **kwargs) 2025-12-04T09:42:12.7778635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7778718Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7778962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7779029Z return func(*args, **kwargs) 2025-12-04T09:42:12.7779267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7779358Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7779595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7779670Z return func(*args, **kwargs) 2025-12-04T09:42:12.7779912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7779993Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7779996Z 2025-12-04T09:42:12.7780107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7780304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7780374Z res = mod(**inputs) 2025-12-04T09:42:12.7780614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7780715Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7780965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7781035Z layer_outputs = layer_module( 2025-12-04T09:42:12.7781255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7781354Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7781620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7781695Z return func(*args, **kwargs) 2025-12-04T09:42:12.7781936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7782014Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7782270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7782337Z return func(*args, **kwargs) 2025-12-04T09:42:12.7782582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7782694Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7782958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7783039Z return func(*args, **kwargs) 2025-12-04T09:42:12.7783302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7783381Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7783385Z 2025-12-04T09:42:12.7783503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7783721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7783806Z res = mod(**inputs) 2025-12-04T09:42:12.7784048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7784120Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7784370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7784441Z layer_outputs = layer_module( 2025-12-04T09:42:12.7784660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7784748Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7784989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7785064Z return func(*args, **kwargs) 2025-12-04T09:42:12.7785306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7785383Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7785627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7785694Z return func(*args, **kwargs) 2025-12-04T09:42:12.7785935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:42:12.7786075Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:42:12.7786080Z 2025-12-04T09:42:12.7786182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7786383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7786448Z res = mod(**inputs) 2025-12-04T09:42:12.7786689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7786789Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7787035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7787112Z layer_outputs = layer_module( 2025-12-04T09:42:12.7787335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7787430Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7787698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7787764Z return func(*args, **kwargs) 2025-12-04T09:42:12.7788005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7788104Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7788345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7788469Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7788709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7788826Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7788830Z 2025-12-04T09:42:12.7788940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7789137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7789208Z res = mod(**inputs) 2025-12-04T09:42:12.7789449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7789519Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7789768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7789837Z layer_outputs = layer_module( 2025-12-04T09:42:12.7790055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7790144Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7790385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7790457Z return func(*args, **kwargs) 2025-12-04T09:42:12.7790699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7790788Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7791034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7791154Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7791414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7791495Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7791499Z 2025-12-04T09:42:12.7791608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7791820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7791884Z res = mod(**inputs) 2025-12-04T09:42:12.7792127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7792205Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7792446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7792520Z layer_outputs = layer_module( 2025-12-04T09:42:12.7792763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7792839Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7793082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7793151Z return func(*args, **kwargs) 2025-12-04T09:42:12.7793406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7793522Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7793761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7793878Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7794117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7794206Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7794209Z 2025-12-04T09:42:12.7794317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7794511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7794600Z res = mod(**inputs) 2025-12-04T09:42:12.7794846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T09:42:12.7794917Z encoder_outputs = self.encoder( 2025-12-04T09:42:12.7795171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7795241Z layer_outputs = layer_module( 2025-12-04T09:42:12.7795480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7795568Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7795824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7795903Z return func(*args, **kwargs) 2025-12-04T09:42:12.7796156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7796249Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7796511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7796632Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7796893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7796972Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7796975Z 2025-12-04T09:42:12.7797075Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7797280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7797342Z res = mod(**inputs) 2025-12-04T09:42:12.7797587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7797668Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7797910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7797988Z layer_outputs = layer_module( 2025-12-04T09:42:12.7798224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7798303Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7798569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7798639Z return func(*args, **kwargs) 2025-12-04T09:42:12.7798909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7799004Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7799260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7799339Z return func(*args, **kwargs) 2025-12-04T09:42:12.7799652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7799757Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7800019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7800089Z return func(*args, **kwargs) 2025-12-04T09:42:12.7800349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7800430Z key_states = self.k(current_states) 2025-12-04T09:42:12.7800434Z 2025-12-04T09:42:12.7800541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7800753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7800847Z res = mod(**inputs) 2025-12-04T09:42:12.7801111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7801195Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7801453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7801533Z layer_outputs = layer_module( 2025-12-04T09:42:12.7801777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7801856Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7802126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7802198Z return func(*args, **kwargs) 2025-12-04T09:42:12.7802455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7802549Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7802817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7802896Z return func(*args, **kwargs) 2025-12-04T09:42:12.7803159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7803246Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7803509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7803581Z return func(*args, **kwargs) 2025-12-04T09:42:12.7803842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7803924Z value_states = self.v(current_states) 2025-12-04T09:42:12.7803929Z 2025-12-04T09:42:12.7804013Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7804103Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7804214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7804422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7804499Z res = mod(**inputs) 2025-12-04T09:42:12.7804754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7804838Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7805115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7805190Z layer_outputs = layer_module( 2025-12-04T09:42:12.7805433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7805514Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7805794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7805887Z return func(*args, **kwargs) 2025-12-04T09:42:12.7806147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7806234Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7806498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7806566Z return func(*args, **kwargs) 2025-12-04T09:42:12.7806833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7806920Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7807182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7807277Z return func(*args, **kwargs) 2025-12-04T09:42:12.7807543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7807633Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7807637Z 2025-12-04T09:42:12.7807747Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7807997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7808072Z res = mod(**inputs) 2025-12-04T09:42:12.7808342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7808427Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7808694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7808771Z layer_outputs = layer_module( 2025-12-04T09:42:12.7809024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7809108Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7809386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7809467Z return func(*args, **kwargs) 2025-12-04T09:42:12.7809741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7809845Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7810126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7810245Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7810509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7810614Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7810619Z 2025-12-04T09:42:12.7810738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7810947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7811014Z res = mod(**inputs) 2025-12-04T09:42:12.7811369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7811458Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7811752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7811843Z layer_outputs = layer_module( 2025-12-04T09:42:12.7812089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7812187Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7812469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7812565Z return func(*args, **kwargs) 2025-12-04T09:42:12.7812836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7812943Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7813198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7813332Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7813587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7813680Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7813703Z 2025-12-04T09:42:12.7813813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7814020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7814098Z res = mod(**inputs) 2025-12-04T09:42:12.7814351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7814437Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7814691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7814765Z layer_outputs = layer_module( 2025-12-04T09:42:12.7815008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7815088Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7815341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7815422Z return func(*args, **kwargs) 2025-12-04T09:42:12.7815681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7815785Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7816038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7816157Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7816415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7816509Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7816512Z 2025-12-04T09:42:12.7816627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7816834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7816904Z res = mod(**inputs) 2025-12-04T09:42:12.7817169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7817248Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7817504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7817585Z layer_outputs = layer_module( 2025-12-04T09:42:12.7817817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7817907Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7818180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7818253Z return func(*args, **kwargs) 2025-12-04T09:42:12.7818513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7818608Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7818876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7819020Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7819270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7819360Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7819363Z 2025-12-04T09:42:12.7819470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7819679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7819753Z res = mod(**inputs) 2025-12-04T09:42:12.7820008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7820112Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7820367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7820442Z layer_outputs = layer_module( 2025-12-04T09:42:12.7820681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7820759Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7821010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7821090Z return func(*args, **kwargs) 2025-12-04T09:42:12.7821338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7821429Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7821682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7821754Z return func(*args, **kwargs) 2025-12-04T09:42:12.7822013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7822101Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7822361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7822429Z return func(*args, **kwargs) 2025-12-04T09:42:12.7822678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7822766Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7822770Z 2025-12-04T09:42:12.7822877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7823080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7823152Z res = mod(**inputs) 2025-12-04T09:42:12.7823392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7823470Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7823707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7823776Z layer_outputs = layer_module( 2025-12-04T09:42:12.7824000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7824093Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7824335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7824411Z return func(*args, **kwargs) 2025-12-04T09:42:12.7824649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7824750Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7824990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7825086Z return func(*args, **kwargs) 2025-12-04T09:42:12.7825331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7825413Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7825660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7825726Z return func(*args, **kwargs) 2025-12-04T09:42:12.7825965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7826067Z key_states = self.k(current_states) 2025-12-04T09:42:12.7826072Z 2025-12-04T09:42:12.7826173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7826369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7826441Z res = mod(**inputs) 2025-12-04T09:42:12.7826678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7826758Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7826996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7827067Z layer_outputs = layer_module( 2025-12-04T09:42:12.7827294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7827369Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7827609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7827683Z return func(*args, **kwargs) 2025-12-04T09:42:12.7827919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7828006Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7828243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7828308Z return func(*args, **kwargs) 2025-12-04T09:42:12.7828551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7828633Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7828876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7828943Z return func(*args, **kwargs) 2025-12-04T09:42:12.7829180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7829260Z value_states = self.v(current_states) 2025-12-04T09:42:12.7829265Z 2025-12-04T09:42:12.7829342Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7829418Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7829528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7829723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7829794Z res = mod(**inputs) 2025-12-04T09:42:12.7830069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7830143Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7830393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7830464Z layer_outputs = layer_module( 2025-12-04T09:42:12.7830698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7830799Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7831041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7831113Z return func(*args, **kwargs) 2025-12-04T09:42:12.7831353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7831430Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7831677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7831745Z return func(*args, **kwargs) 2025-12-04T09:42:12.7831983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7832088Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7832326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7832400Z return func(*args, **kwargs) 2025-12-04T09:42:12.7832638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7832713Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7832717Z 2025-12-04T09:42:12.7832828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7833022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7833093Z res = mod(**inputs) 2025-12-04T09:42:12.7833332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7833405Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7833652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7833724Z layer_outputs = layer_module( 2025-12-04T09:42:12.7833943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7834026Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7834264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7834337Z return func(*args, **kwargs) 2025-12-04T09:42:12.7834575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7834654Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7834899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7834967Z return func(*args, **kwargs) 2025-12-04T09:42:12.7835203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7835295Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7835532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7835606Z return func(*args, **kwargs) 2025-12-04T09:42:12.7835841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7835940Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7835945Z 2025-12-04T09:42:12.7836057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7836254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7836325Z res = mod(**inputs) 2025-12-04T09:42:12.7836603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7836692Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7836943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7837013Z layer_outputs = layer_module( 2025-12-04T09:42:12.7837236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7837318Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7837571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7837643Z return func(*args, **kwargs) 2025-12-04T09:42:12.7837879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7837977Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7838218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7838285Z return func(*args, **kwargs) 2025-12-04T09:42:12.7838514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7838599Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7838828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7838899Z return func(*args, **kwargs) 2025-12-04T09:42:12.7839128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7839202Z key_states = self.k(current_states) 2025-12-04T09:42:12.7839207Z 2025-12-04T09:42:12.7839314Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7839504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7839572Z res = mod(**inputs) 2025-12-04T09:42:12.7839805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7839872Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7840118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7840187Z layer_outputs = layer_module( 2025-12-04T09:42:12.7840408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7840489Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7840727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7840802Z return func(*args, **kwargs) 2025-12-04T09:42:12.7841039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7841117Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7841367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7841432Z return func(*args, **kwargs) 2025-12-04T09:42:12.7841676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7841776Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7842015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7842090Z return func(*args, **kwargs) 2025-12-04T09:42:12.7842325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7842419Z value_states = self.v(current_states) 2025-12-04T09:42:12.7842438Z 2025-12-04T09:42:12.7842528Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7842606Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7842714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7842911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7842972Z res = mod(**inputs) 2025-12-04T09:42:12.7843224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7843296Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7843539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7843654Z layer_outputs = layer_module( 2025-12-04T09:42:12.7843880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7843967Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7844248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7844328Z return func(*args, **kwargs) 2025-12-04T09:42:12.7844571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7844649Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7844893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7844971Z return func(*args, **kwargs) 2025-12-04T09:42:12.7845213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7845306Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7845551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7845621Z return func(*args, **kwargs) 2025-12-04T09:42:12.7845866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7845943Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7845947Z 2025-12-04T09:42:12.7846059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7846255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7846320Z res = mod(**inputs) 2025-12-04T09:42:12.7846571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7846647Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7846895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7846976Z layer_outputs = layer_module( 2025-12-04T09:42:12.7847204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7847291Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7847537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7847606Z return func(*args, **kwargs) 2025-12-04T09:42:12.7847870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7847966Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7848204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7848331Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7848602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7848731Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7848735Z 2025-12-04T09:42:12.7848841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7849079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7849157Z res = mod(**inputs) 2025-12-04T09:42:12.7849419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7849504Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7849760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7849854Z layer_outputs = layer_module( 2025-12-04T09:42:12.7850098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7850178Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7850439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7850518Z return func(*args, **kwargs) 2025-12-04T09:42:12.7850779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7850884Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7851133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7851251Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7851602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7851696Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7851702Z 2025-12-04T09:42:12.7851819Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7852041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7852108Z res = mod(**inputs) 2025-12-04T09:42:12.7852382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7852459Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7852736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7852823Z layer_outputs = layer_module( 2025-12-04T09:42:12.7853067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7853152Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7853396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7853464Z return func(*args, **kwargs) 2025-12-04T09:42:12.7853733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7853827Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7854090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7854243Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7854503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7854605Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7854609Z 2025-12-04T09:42:12.7854721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7854952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7855046Z res = mod(**inputs) 2025-12-04T09:42:12.7855309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7855395Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7855656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7855732Z layer_outputs = layer_module( 2025-12-04T09:42:12.7855980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7856062Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7856342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7856422Z return func(*args, **kwargs) 2025-12-04T09:42:12.7856680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7856782Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7857039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7857159Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7857425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7857508Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7857512Z 2025-12-04T09:42:12.7857628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7857841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7857908Z res = mod(**inputs) 2025-12-04T09:42:12.7858177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7858256Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7858514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7858596Z layer_outputs = layer_module( 2025-12-04T09:42:12.7858834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7858926Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7859187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7859259Z return func(*args, **kwargs) 2025-12-04T09:42:12.7859527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7859613Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7859872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7859954Z return func(*args, **kwargs) 2025-12-04T09:42:12.7860211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7860306Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7860588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7860662Z return func(*args, **kwargs) 2025-12-04T09:42:12.7860940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7861017Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7861021Z 2025-12-04T09:42:12.7861151Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7861350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7861432Z res = mod(**inputs) 2025-12-04T09:42:12.7861684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7861755Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7861995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7862075Z layer_outputs = layer_module( 2025-12-04T09:42:12.7862296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7862378Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7862641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7862710Z return func(*args, **kwargs) 2025-12-04T09:42:12.7862957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7863037Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7863273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7863347Z return func(*args, **kwargs) 2025-12-04T09:42:12.7863584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7863674Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7863911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7863979Z return func(*args, **kwargs) 2025-12-04T09:42:12.7864225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7864302Z key_states = self.k(current_states) 2025-12-04T09:42:12.7864306Z 2025-12-04T09:42:12.7864416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7864609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7864670Z res = mod(**inputs) 2025-12-04T09:42:12.7864915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7864988Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7865227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7865304Z layer_outputs = layer_module( 2025-12-04T09:42:12.7865524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7865607Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7865848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7865912Z return func(*args, **kwargs) 2025-12-04T09:42:12.7866155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7866232Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7866487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7866559Z return func(*args, **kwargs) 2025-12-04T09:42:12.7866794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7866879Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7867131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7867215Z return func(*args, **kwargs) 2025-12-04T09:42:12.7867468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7867543Z value_states = self.v(current_states) 2025-12-04T09:42:12.7867547Z 2025-12-04T09:42:12.7867634Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7867710Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7867813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7868021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7868083Z res = mod(**inputs) 2025-12-04T09:42:12.7868332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7868430Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7868671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7868749Z layer_outputs = layer_module( 2025-12-04T09:42:12.7868970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7869049Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7869294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7869363Z return func(*args, **kwargs) 2025-12-04T09:42:12.7869597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7869682Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7869918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7869993Z return func(*args, **kwargs) 2025-12-04T09:42:12.7870229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7870310Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7870556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7870623Z return func(*args, **kwargs) 2025-12-04T09:42:12.7870869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7870943Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7870947Z 2025-12-04T09:42:12.7871048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7871248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7871314Z res = mod(**inputs) 2025-12-04T09:42:12.7871557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7871637Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7871875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7871952Z layer_outputs = layer_module( 2025-12-04T09:42:12.7872170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7872394Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7872669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7872738Z return func(*args, **kwargs) 2025-12-04T09:42:12.7872981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7873113Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7873357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7873488Z return func(*args, **kwargs) 2025-12-04T09:42:12.7873730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:42:12.7873862Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:42:12.7873866Z 2025-12-04T09:42:12.7873981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7874176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7874253Z res = mod(**inputs) 2025-12-04T09:42:12.7874505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7874608Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7874861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7874934Z layer_outputs = layer_module( 2025-12-04T09:42:12.7875152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7875241Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7875491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7875566Z return func(*args, **kwargs) 2025-12-04T09:42:12.7875800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7875877Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7876122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7876189Z return func(*args, **kwargs) 2025-12-04T09:42:12.7876423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7876512Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7876745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7876817Z return func(*args, **kwargs) 2025-12-04T09:42:12.7877047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7877121Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7877125Z 2025-12-04T09:42:12.7877233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7877423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7877490Z res = mod(**inputs) 2025-12-04T09:42:12.7877726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7877799Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7878038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7878106Z layer_outputs = layer_module( 2025-12-04T09:42:12.7878319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7878422Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7878658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7878731Z return func(*args, **kwargs) 2025-12-04T09:42:12.7878964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7879057Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7879313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7879377Z return func(*args, **kwargs) 2025-12-04T09:42:12.7879603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7879690Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7879922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7879993Z return func(*args, **kwargs) 2025-12-04T09:42:12.7880223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7880314Z key_states = self.k(current_states) 2025-12-04T09:42:12.7880318Z 2025-12-04T09:42:12.7880426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7880617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7880686Z res = mod(**inputs) 2025-12-04T09:42:12.7880925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7880994Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7881242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7881312Z layer_outputs = layer_module( 2025-12-04T09:42:12.7881534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7881614Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7881863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7881937Z return func(*args, **kwargs) 2025-12-04T09:42:12.7882171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7882247Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7882489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7882553Z return func(*args, **kwargs) 2025-12-04T09:42:12.7882789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7882875Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7883113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7883188Z return func(*args, **kwargs) 2025-12-04T09:42:12.7883430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7883509Z value_states = self.v(current_states) 2025-12-04T09:42:12.7883513Z 2025-12-04T09:42:12.7883600Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7883678Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7883788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7883983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7884047Z res = mod(**inputs) 2025-12-04T09:42:12.7884314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7884387Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7884626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7884706Z layer_outputs = layer_module( 2025-12-04T09:42:12.7884941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7885043Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7885286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7885357Z return func(*args, **kwargs) 2025-12-04T09:42:12.7885621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7885710Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7885964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7886046Z return func(*args, **kwargs) 2025-12-04T09:42:12.7886331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7886426Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7886679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7886748Z return func(*args, **kwargs) 2025-12-04T09:42:12.7887007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7887084Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7887088Z 2025-12-04T09:42:12.7887201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7887418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7887483Z res = mod(**inputs) 2025-12-04T09:42:12.7887744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7887822Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7888075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7888159Z layer_outputs = layer_module( 2025-12-04T09:42:12.7888391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7888474Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7888726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7888792Z return func(*args, **kwargs) 2025-12-04T09:42:12.7889038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7889132Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7889390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7889520Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7889770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7889880Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7889884Z 2025-12-04T09:42:12.7889992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7890197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7890271Z res = mod(**inputs) 2025-12-04T09:42:12.7890543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7890629Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7890886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7890960Z layer_outputs = layer_module( 2025-12-04T09:42:12.7891223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7891519Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7891803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7891884Z return func(*args, **kwargs) 2025-12-04T09:42:12.7892148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7892259Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7892534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7892683Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7892945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7893029Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7893033Z 2025-12-04T09:42:12.7893147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7893364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7893431Z res = mod(**inputs) 2025-12-04T09:42:12.7893694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7893771Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7894026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7894107Z layer_outputs = layer_module( 2025-12-04T09:42:12.7894340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7894428Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7894680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7894750Z return func(*args, **kwargs) 2025-12-04T09:42:12.7895028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7895119Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7895372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7895497Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7895748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7895846Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7895850Z 2025-12-04T09:42:12.7895955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7896161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7896234Z res = mod(**inputs) 2025-12-04T09:42:12.7896485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7896566Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7896819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7896907Z layer_outputs = layer_module( 2025-12-04T09:42:12.7897149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7897227Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7897480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7897576Z return func(*args, **kwargs) 2025-12-04T09:42:12.7897845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7897943Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7898191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7898307Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7898564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7898646Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7898650Z 2025-12-04T09:42:12.7898762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7898997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7899065Z res = mod(**inputs) 2025-12-04T09:42:12.7899329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7899405Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7899658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7899738Z layer_outputs = layer_module( 2025-12-04T09:42:12.7899971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7900059Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7900313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7900385Z return func(*args, **kwargs) 2025-12-04T09:42:12.7900645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7900727Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7900980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7901056Z return func(*args, **kwargs) 2025-12-04T09:42:12.7901306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7901398Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7901654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7901724Z return func(*args, **kwargs) 2025-12-04T09:42:12.7901980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7902063Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7902067Z 2025-12-04T09:42:12.7902182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7902386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7902450Z res = mod(**inputs) 2025-12-04T09:42:12.7902710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7902784Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7903053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7903135Z layer_outputs = layer_module( 2025-12-04T09:42:12.7903375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7903465Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7903737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7903820Z return func(*args, **kwargs) 2025-12-04T09:42:12.7904068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7904145Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7904388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7904460Z return func(*args, **kwargs) 2025-12-04T09:42:12.7904699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7904788Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7905024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7905110Z return func(*args, **kwargs) 2025-12-04T09:42:12.7905357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7905436Z key_states = self.k(current_states) 2025-12-04T09:42:12.7905439Z 2025-12-04T09:42:12.7905548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7905746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7905808Z res = mod(**inputs) 2025-12-04T09:42:12.7906059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7906131Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7906370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7906450Z layer_outputs = layer_module( 2025-12-04T09:42:12.7906672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7906756Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7906997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7907065Z return func(*args, **kwargs) 2025-12-04T09:42:12.7907314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7907394Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7907639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7907715Z return func(*args, **kwargs) 2025-12-04T09:42:12.7907955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7908044Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7908288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7908355Z return func(*args, **kwargs) 2025-12-04T09:42:12.7908601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7908677Z value_states = self.v(current_states) 2025-12-04T09:42:12.7908680Z 2025-12-04T09:42:12.7908764Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7908841Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7908963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7909169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7909232Z res = mod(**inputs) 2025-12-04T09:42:12.7909478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7909558Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7909821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7909918Z layer_outputs = layer_module( 2025-12-04T09:42:12.7910142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7910219Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7910470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7910537Z return func(*args, **kwargs) 2025-12-04T09:42:12.7910788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7910898Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7911163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7911240Z return func(*args, **kwargs) 2025-12-04T09:42:12.7911495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7911579Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7911842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7911919Z return func(*args, **kwargs) 2025-12-04T09:42:12.7912161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7912243Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7912246Z 2025-12-04T09:42:12.7912346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7912552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7912613Z res = mod(**inputs) 2025-12-04T09:42:12.7912859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7912941Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7913182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7913258Z layer_outputs = layer_module( 2025-12-04T09:42:12.7913480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7913558Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7913803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7913869Z return func(*args, **kwargs) 2025-12-04T09:42:12.7914108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7914197Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7914438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7914512Z return func(*args, **kwargs) 2025-12-04T09:42:12.7914753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7914835Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7915101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7915169Z return func(*args, **kwargs) 2025-12-04T09:42:12.7915420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7915500Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7915504Z 2025-12-04T09:42:12.7915623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7915854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7915916Z res = mod(**inputs) 2025-12-04T09:42:12.7916158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7916236Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7916478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7916557Z layer_outputs = layer_module( 2025-12-04T09:42:12.7916785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7916861Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7917120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7917184Z return func(*args, **kwargs) 2025-12-04T09:42:12.7917425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7917517Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7917770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7917848Z return func(*args, **kwargs) 2025-12-04T09:42:12.7918100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7918185Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7918446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7918516Z return func(*args, **kwargs) 2025-12-04T09:42:12.7918774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7918853Z key_states = self.k(current_states) 2025-12-04T09:42:12.7918857Z 2025-12-04T09:42:12.7918965Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7919175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7919247Z res = mod(**inputs) 2025-12-04T09:42:12.7919485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7919563Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7919801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7919876Z layer_outputs = layer_module( 2025-12-04T09:42:12.7920098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7920173Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7920421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7920490Z return func(*args, **kwargs) 2025-12-04T09:42:12.7920727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7920811Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7921068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7921143Z return func(*args, **kwargs) 2025-12-04T09:42:12.7921381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7921464Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7921725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7921810Z return func(*args, **kwargs) 2025-12-04T09:42:12.7922058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7922136Z value_states = self.v(current_states) 2025-12-04T09:42:12.7922139Z 2025-12-04T09:42:12.7922218Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7922306Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7922408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7922603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7922672Z res = mod(**inputs) 2025-12-04T09:42:12.7922921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7923020Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7923265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7923335Z layer_outputs = layer_module( 2025-12-04T09:42:12.7923561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7923636Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7923878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7923954Z return func(*args, **kwargs) 2025-12-04T09:42:12.7924193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7924279Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7924524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7924590Z return func(*args, **kwargs) 2025-12-04T09:42:12.7924842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7924922Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7925161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7925234Z return func(*args, **kwargs) 2025-12-04T09:42:12.7925477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7925563Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7925566Z 2025-12-04T09:42:12.7925667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7925865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7925934Z res = mod(**inputs) 2025-12-04T09:42:12.7926177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7926253Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7926496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7926565Z layer_outputs = layer_module( 2025-12-04T09:42:12.7926793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7926883Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7927128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7927204Z return func(*args, **kwargs) 2025-12-04T09:42:12.7927442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7927543Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7927798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7927864Z return func(*args, **kwargs) 2025-12-04T09:42:12.7928114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 529, in forward 2025-12-04T09:42:12.7928249Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:42:12.7928253Z 2025-12-04T09:42:12.7928370Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7928580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7928643Z res = mod(**inputs) 2025-12-04T09:42:12.7928927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7929002Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7929257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7929338Z layer_outputs = layer_module( 2025-12-04T09:42:12.7929580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7929666Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7929929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7929999Z return func(*args, **kwargs) 2025-12-04T09:42:12.7930261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7930355Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7930610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7930741Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7930992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7931102Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7931105Z 2025-12-04T09:42:12.7931213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7931503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7931588Z res = mod(**inputs) 2025-12-04T09:42:12.7931844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7931930Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7932197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7932275Z layer_outputs = layer_module( 2025-12-04T09:42:12.7932529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7932613Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7932884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7932966Z return func(*args, **kwargs) 2025-12-04T09:42:12.7933284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7933408Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7933669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7933792Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7934069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7934170Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7934174Z 2025-12-04T09:42:12.7934286Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7934508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7934573Z res = mod(**inputs) 2025-12-04T09:42:12.7934833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7934908Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7935160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7935262Z layer_outputs = layer_module( 2025-12-04T09:42:12.7935492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7935581Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7935834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7935903Z return func(*args, **kwargs) 2025-12-04T09:42:12.7936165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7936259Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7936511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7936637Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7936886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7936985Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7936988Z 2025-12-04T09:42:12.7937097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7937304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7937377Z res = mod(**inputs) 2025-12-04T09:42:12.7937629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7937710Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7937964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7938034Z layer_outputs = layer_module( 2025-12-04T09:42:12.7938274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7938355Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7938610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7938688Z return func(*args, **kwargs) 2025-12-04T09:42:12.7938937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7939036Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7939285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7939401Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7939671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7939756Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7939761Z 2025-12-04T09:42:12.7939874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7940105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7940173Z res = mod(**inputs) 2025-12-04T09:42:12.7940459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7940533Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7940789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7940867Z layer_outputs = layer_module( 2025-12-04T09:42:12.7941103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7941188Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7941442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7941531Z return func(*args, **kwargs) 2025-12-04T09:42:12.7941795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7941879Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7942133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7942208Z return func(*args, **kwargs) 2025-12-04T09:42:12.7942474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7942563Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7942806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7942869Z return func(*args, **kwargs) 2025-12-04T09:42:12.7943119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7943197Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7943203Z 2025-12-04T09:42:12.7943316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7943514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7943576Z res = mod(**inputs) 2025-12-04T09:42:12.7943823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7943893Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7944137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7944213Z layer_outputs = layer_module( 2025-12-04T09:42:12.7944433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7944519Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7944763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7944832Z return func(*args, **kwargs) 2025-12-04T09:42:12.7945078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7945156Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7945397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7945471Z return func(*args, **kwargs) 2025-12-04T09:42:12.7945732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7945828Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7946070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7946142Z return func(*args, **kwargs) 2025-12-04T09:42:12.7946414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7946506Z key_states = self.k(current_states) 2025-12-04T09:42:12.7946510Z 2025-12-04T09:42:12.7946623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7946820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7946883Z res = mod(**inputs) 2025-12-04T09:42:12.7947134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7947205Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7947446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7947545Z layer_outputs = layer_module( 2025-12-04T09:42:12.7947769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7947855Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7948096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7948161Z return func(*args, **kwargs) 2025-12-04T09:42:12.7948407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7948484Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7948724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7948799Z return func(*args, **kwargs) 2025-12-04T09:42:12.7949039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7949127Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7949370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7949436Z return func(*args, **kwargs) 2025-12-04T09:42:12.7949685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7949761Z value_states = self.v(current_states) 2025-12-04T09:42:12.7949764Z 2025-12-04T09:42:12.7949848Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7949923Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7950025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7950229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7950291Z res = mod(**inputs) 2025-12-04T09:42:12.7950535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7950615Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7950858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7950936Z layer_outputs = layer_module( 2025-12-04T09:42:12.7951158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7951236Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7951499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7951570Z return func(*args, **kwargs) 2025-12-04T09:42:12.7951808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7951897Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7952150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7952242Z return func(*args, **kwargs) 2025-12-04T09:42:12.7952482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7952564Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7952812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7952878Z return func(*args, **kwargs) 2025-12-04T09:42:12.7953118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7953200Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7953204Z 2025-12-04T09:42:12.7953324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7953525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7953587Z res = mod(**inputs) 2025-12-04T09:42:12.7953833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7953909Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7954151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7954226Z layer_outputs = layer_module( 2025-12-04T09:42:12.7954445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7954521Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7954768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7954834Z return func(*args, **kwargs) 2025-12-04T09:42:12.7955072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7955157Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7955393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7955466Z return func(*args, **kwargs) 2025-12-04T09:42:12.7955702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7955785Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7956031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7956095Z return func(*args, **kwargs) 2025-12-04T09:42:12.7956331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7956413Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7956417Z 2025-12-04T09:42:12.7956521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7956726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7956788Z res = mod(**inputs) 2025-12-04T09:42:12.7957031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7957109Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7957367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7957449Z layer_outputs = layer_module( 2025-12-04T09:42:12.7957672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7957750Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7958015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7958100Z return func(*args, **kwargs) 2025-12-04T09:42:12.7958340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7958425Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7958664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7958736Z return func(*args, **kwargs) 2025-12-04T09:42:12.7958977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7959059Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7959306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7959398Z return func(*args, **kwargs) 2025-12-04T09:42:12.7959646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7959725Z key_states = self.k(current_states) 2025-12-04T09:42:12.7959728Z 2025-12-04T09:42:12.7959833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7960035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7960097Z res = mod(**inputs) 2025-12-04T09:42:12.7960342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7960421Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7960662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7960743Z layer_outputs = layer_module( 2025-12-04T09:42:12.7960966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7961043Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7961289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7961354Z return func(*args, **kwargs) 2025-12-04T09:42:12.7961596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7961682Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7961923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7961995Z return func(*args, **kwargs) 2025-12-04T09:42:12.7962233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7962315Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7962564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7962632Z return func(*args, **kwargs) 2025-12-04T09:42:12.7962879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7962954Z value_states = self.v(current_states) 2025-12-04T09:42:12.7962957Z 2025-12-04T09:42:12.7963034Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7963116Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7963244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7963438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7963507Z res = mod(**inputs) 2025-12-04T09:42:12.7963751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7963846Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7964100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7964169Z layer_outputs = layer_module( 2025-12-04T09:42:12.7964404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7964482Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7964733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7964809Z return func(*args, **kwargs) 2025-12-04T09:42:12.7965058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7965170Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7965424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7965494Z return func(*args, **kwargs) 2025-12-04T09:42:12.7965751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7965835Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7966084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7966162Z return func(*args, **kwargs) 2025-12-04T09:42:12.7966412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7966500Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7966503Z 2025-12-04T09:42:12.7966612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7966817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7966890Z res = mod(**inputs) 2025-12-04T09:42:12.7967145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7967224Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7967475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7967548Z layer_outputs = layer_module( 2025-12-04T09:42:12.7967786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7967864Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7968123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7968201Z return func(*args, **kwargs) 2025-12-04T09:42:12.7968452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7968552Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7968817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7968941Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7969202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.7969306Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.7969329Z 2025-12-04T09:42:12.7969450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7969684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7969754Z res = mod(**inputs) 2025-12-04T09:42:12.7970024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7970117Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7970395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7970478Z layer_outputs = layer_module( 2025-12-04T09:42:12.7970715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7970803Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7971073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7971143Z return func(*args, **kwargs) 2025-12-04T09:42:12.7971483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7971608Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7971871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7972001Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7972253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.7972488Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.7972494Z 2025-12-04T09:42:12.7972607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7972835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7972909Z res = mod(**inputs) 2025-12-04T09:42:12.7973163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7973462Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7973717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7973792Z layer_outputs = layer_module( 2025-12-04T09:42:12.7974032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7974111Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7974373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7974452Z return func(*args, **kwargs) 2025-12-04T09:42:12.7974715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7974817Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7975074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7975197Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7975470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.7975563Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.7975567Z 2025-12-04T09:42:12.7975682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7975903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7975970Z res = mod(**inputs) 2025-12-04T09:42:12.7976279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7976357Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7976614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7976699Z layer_outputs = layer_module( 2025-12-04T09:42:12.7976971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7977089Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7977345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7977418Z return func(*args, **kwargs) 2025-12-04T09:42:12.7977680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7977786Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7978031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.7978143Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.7978396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.7978479Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.7978484Z 2025-12-04T09:42:12.7978583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7978771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7978840Z res = mod(**inputs) 2025-12-04T09:42:12.7979077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7979154Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7979385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7979453Z layer_outputs = layer_module( 2025-12-04T09:42:12.7979676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7979751Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7979985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7980058Z return func(*args, **kwargs) 2025-12-04T09:42:12.7980289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.7980380Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.7980611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 218, in forward 2025-12-04T09:42:12.7980735Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-12-04T09:42:12.7980739Z 2025-12-04T09:42:12.7980844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7981033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7981104Z res = mod(**inputs) 2025-12-04T09:42:12.7981339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7981408Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7981651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7981717Z layer_outputs = layer_module( 2025-12-04T09:42:12.7981930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7982013Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7982266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7982340Z return func(*args, **kwargs) 2025-12-04T09:42:12.7982572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7982652Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7982906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7982988Z return func(*args, **kwargs) 2025-12-04T09:42:12.7983231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7983311Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7983547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7983619Z return func(*args, **kwargs) 2025-12-04T09:42:12.7983856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7983930Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7983960Z 2025-12-04T09:42:12.7984070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7984262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7984331Z res = mod(**inputs) 2025-12-04T09:42:12.7984567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7984636Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7984877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7984942Z layer_outputs = layer_module( 2025-12-04T09:42:12.7985155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7985234Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7985464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7985538Z return func(*args, **kwargs) 2025-12-04T09:42:12.7985769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7985847Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7986088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7986151Z return func(*args, **kwargs) 2025-12-04T09:42:12.7986389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7986470Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7986704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7986775Z return func(*args, **kwargs) 2025-12-04T09:42:12.7987005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7987079Z key_states = self.k(current_states) 2025-12-04T09:42:12.7987084Z 2025-12-04T09:42:12.7987191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7987380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7987449Z res = mod(**inputs) 2025-12-04T09:42:12.7987684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7987753Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7988015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7988086Z layer_outputs = layer_module( 2025-12-04T09:42:12.7988297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7988380Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7988632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7988723Z return func(*args, **kwargs) 2025-12-04T09:42:12.7988961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7989037Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7989278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7989344Z return func(*args, **kwargs) 2025-12-04T09:42:12.7989584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7989662Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7989912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7989986Z return func(*args, **kwargs) 2025-12-04T09:42:12.7990216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.7990289Z value_states = self.v(current_states) 2025-12-04T09:42:12.7990293Z 2025-12-04T09:42:12.7990377Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7990450Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.7990556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7990745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7990805Z res = mod(**inputs) 2025-12-04T09:42:12.7991044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7991116Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7991349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7991426Z layer_outputs = layer_module( 2025-12-04T09:42:12.7991638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7991721Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7991963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7992027Z return func(*args, **kwargs) 2025-12-04T09:42:12.7992266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.7992343Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.7992573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7992647Z return func(*args, **kwargs) 2025-12-04T09:42:12.7992878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.7992962Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.7993194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7993258Z return func(*args, **kwargs) 2025-12-04T09:42:12.7993496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.7993585Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.7993589Z 2025-12-04T09:42:12.7993696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7993884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7993945Z res = mod(**inputs) 2025-12-04T09:42:12.7994210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7994297Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7994531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7994602Z layer_outputs = layer_module( 2025-12-04T09:42:12.7994817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7994898Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7995139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7995206Z return func(*args, **kwargs) 2025-12-04T09:42:12.7995448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7995545Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7995779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7995852Z return func(*args, **kwargs) 2025-12-04T09:42:12.7996083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7996171Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7996403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7996468Z return func(*args, **kwargs) 2025-12-04T09:42:12.7996706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.7996779Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.7996784Z 2025-12-04T09:42:12.7996887Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.7997074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.7997137Z res = mod(**inputs) 2025-12-04T09:42:12.7997378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.7997446Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.7997678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.7997753Z layer_outputs = layer_module( 2025-12-04T09:42:12.7997965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.7998045Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.7998275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7998339Z return func(*args, **kwargs) 2025-12-04T09:42:12.7998576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.7998652Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.7998889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7998953Z return func(*args, **kwargs) 2025-12-04T09:42:12.7999181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.7999280Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.7999517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.7999580Z return func(*args, **kwargs) 2025-12-04T09:42:12.7999821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.7999909Z key_states = self.k(current_states) 2025-12-04T09:42:12.7999929Z 2025-12-04T09:42:12.8000036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8000226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8000287Z res = mod(**inputs) 2025-12-04T09:42:12.8000532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8000601Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8000837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8000912Z layer_outputs = layer_module( 2025-12-04T09:42:12.8001127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8001225Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8001456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8001522Z return func(*args, **kwargs) 2025-12-04T09:42:12.8001759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8001837Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8002079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8002145Z return func(*args, **kwargs) 2025-12-04T09:42:12.8002382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8002469Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8002708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8002776Z return func(*args, **kwargs) 2025-12-04T09:42:12.8003023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.8003101Z value_states = self.v(current_states) 2025-12-04T09:42:12.8003104Z 2025-12-04T09:42:12.8003189Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8003265Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8003364Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8003567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8003631Z res = mod(**inputs) 2025-12-04T09:42:12.8003873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8003953Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8004195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8004272Z layer_outputs = layer_module( 2025-12-04T09:42:12.8004491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8004566Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8004814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8004881Z return func(*args, **kwargs) 2025-12-04T09:42:12.8005137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8005228Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8005470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8005546Z return func(*args, **kwargs) 2025-12-04T09:42:12.8005803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8005915Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8006173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8006245Z return func(*args, **kwargs) 2025-12-04T09:42:12.8006503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.8006584Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.8006589Z 2025-12-04T09:42:12.8006697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8006909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8006993Z res = mod(**inputs) 2025-12-04T09:42:12.8007257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8007340Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8007603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8007683Z layer_outputs = layer_module( 2025-12-04T09:42:12.8007923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8008000Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8008269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8008339Z return func(*args, **kwargs) 2025-12-04T09:42:12.8008599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8008703Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8008964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8009095Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8009355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.8009459Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.8009463Z 2025-12-04T09:42:12.8009580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8009791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8009865Z res = mod(**inputs) 2025-12-04T09:42:12.8010131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8010208Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8010481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8010558Z layer_outputs = layer_module( 2025-12-04T09:42:12.8010797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8010883Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8011145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8011223Z return func(*args, **kwargs) 2025-12-04T09:42:12.8011578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8011682Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8011942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8012065Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8012348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.8012454Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.8012458Z 2025-12-04T09:42:12.8012567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8012787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8012867Z res = mod(**inputs) 2025-12-04T09:42:12.8013123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8013208Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8013461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8013566Z layer_outputs = layer_module( 2025-12-04T09:42:12.8013806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8013887Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8014156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8014226Z return func(*args, **kwargs) 2025-12-04T09:42:12.8014486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8014592Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8014851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8014979Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8015243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.8015335Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.8015341Z 2025-12-04T09:42:12.8015455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8015668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8015740Z res = mod(**inputs) 2025-12-04T09:42:12.8016003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8016076Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8016351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8016425Z layer_outputs = layer_module( 2025-12-04T09:42:12.8016665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8016754Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8017016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8017095Z return func(*args, **kwargs) 2025-12-04T09:42:12.8017356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8017449Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8017719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8017857Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8018116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.8018201Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.8018205Z 2025-12-04T09:42:12.8018313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8018545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8018631Z res = mod(**inputs) 2025-12-04T09:42:12.8018887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8018972Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8019233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8019315Z layer_outputs = layer_module( 2025-12-04T09:42:12.8019550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8019634Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8019917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8019989Z return func(*args, **kwargs) 2025-12-04T09:42:12.8020239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8020333Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8020585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8020663Z return func(*args, **kwargs) 2025-12-04T09:42:12.8020910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.8020998Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.8021260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8021331Z return func(*args, **kwargs) 2025-12-04T09:42:12.8021590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.8021671Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.8021676Z 2025-12-04T09:42:12.8021784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8021995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8022060Z res = mod(**inputs) 2025-12-04T09:42:12.8022314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8022394Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8022651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8022730Z layer_outputs = layer_module( 2025-12-04T09:42:12.8022962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8023041Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8023306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8023376Z return func(*args, **kwargs) 2025-12-04T09:42:12.8023610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8023697Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8023933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8024026Z return func(*args, **kwargs) 2025-12-04T09:42:12.8024263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.8024343Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.8024588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8024669Z return func(*args, **kwargs) 2025-12-04T09:42:12.8024932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.8025007Z key_states = self.k(current_states) 2025-12-04T09:42:12.8025011Z 2025-12-04T09:42:12.8025115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8025315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8025379Z res = mod(**inputs) 2025-12-04T09:42:12.8025619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8025695Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8025934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8026048Z layer_outputs = layer_module( 2025-12-04T09:42:12.8026275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8026352Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8026605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8026671Z return func(*args, **kwargs) 2025-12-04T09:42:12.8026910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8026996Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8027237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8027312Z return func(*args, **kwargs) 2025-12-04T09:42:12.8027551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.8027632Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.8027881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8027947Z return func(*args, **kwargs) 2025-12-04T09:42:12.8028195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.8028270Z value_states = self.v(current_states) 2025-12-04T09:42:12.8028274Z 2025-12-04T09:42:12.8028355Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8028439Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8028540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8028737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8028810Z res = mod(**inputs) 2025-12-04T09:42:12.8029053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8029131Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8029374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8029445Z layer_outputs = layer_module( 2025-12-04T09:42:12.8029676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8029752Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8030013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8030088Z return func(*args, **kwargs) 2025-12-04T09:42:12.8030329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8030416Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8030683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8030765Z return func(*args, **kwargs) 2025-12-04T09:42:12.8031010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.8031089Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.8031338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8031404Z return func(*args, **kwargs) 2025-12-04T09:42:12.8031640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.8031722Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.8031742Z 2025-12-04T09:42:12.8031847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8032047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8032118Z res = mod(**inputs) 2025-12-04T09:42:12.8032362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8032441Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8032686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8032755Z layer_outputs = layer_module( 2025-12-04T09:42:12.8032990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8033065Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8033313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8033392Z return func(*args, **kwargs) 2025-12-04T09:42:12.8033643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8033731Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8033979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8034046Z return func(*args, **kwargs) 2025-12-04T09:42:12.8034300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T09:42:12.8034436Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:42:12.8034440Z 2025-12-04T09:42:12.8034549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8034748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8034814Z res = mod(**inputs) 2025-12-04T09:42:12.8035073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8035145Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8035394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8035471Z layer_outputs = layer_module( 2025-12-04T09:42:12.8035697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8035777Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8036088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8036154Z return func(*args, **kwargs) 2025-12-04T09:42:12.8036395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8036474Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8036727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8036816Z return func(*args, **kwargs) 2025-12-04T09:42:12.8037052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8037140Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8037374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8037441Z return func(*args, **kwargs) 2025-12-04T09:42:12.8037679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.8037772Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.8037776Z 2025-12-04T09:42:12.8037882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8038073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8038137Z res = mod(**inputs) 2025-12-04T09:42:12.8038378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8038448Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8038681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8038756Z layer_outputs = layer_module( 2025-12-04T09:42:12.8038971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8039050Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8039282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8039349Z return func(*args, **kwargs) 2025-12-04T09:42:12.8039593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8039670Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8039901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8039974Z return func(*args, **kwargs) 2025-12-04T09:42:12.8040205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8040293Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8040524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8040591Z return func(*args, **kwargs) 2025-12-04T09:42:12.8040829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.8040904Z key_states = self.k(current_states) 2025-12-04T09:42:12.8040909Z 2025-12-04T09:42:12.8041013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8041197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8041258Z res = mod(**inputs) 2025-12-04T09:42:12.8041497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8041566Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8041814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8041891Z layer_outputs = layer_module( 2025-12-04T09:42:12.8042105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8042187Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8042433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8042513Z return func(*args, **kwargs) 2025-12-04T09:42:12.8042759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8042833Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8043072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8043138Z return func(*args, **kwargs) 2025-12-04T09:42:12.8043370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8043457Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8043707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8043773Z return func(*args, **kwargs) 2025-12-04T09:42:12.8044015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.8044087Z value_states = self.v(current_states) 2025-12-04T09:42:12.8044090Z 2025-12-04T09:42:12.8044172Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8044246Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8044345Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8044541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8044601Z res = mod(**inputs) 2025-12-04T09:42:12.8044836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8044914Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8045150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8045224Z layer_outputs = layer_module( 2025-12-04T09:42:12.8045440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8045514Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8045752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8045816Z return func(*args, **kwargs) 2025-12-04T09:42:12.8046048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8046132Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8046362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8046435Z return func(*args, **kwargs) 2025-12-04T09:42:12.8046667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8046745Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8046985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8047048Z return func(*args, **kwargs) 2025-12-04T09:42:12.8047284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.8047375Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.8047379Z 2025-12-04T09:42:12.8047480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8047678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8047742Z res = mod(**inputs) 2025-12-04T09:42:12.8047991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8048097Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8048334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8048409Z layer_outputs = layer_module( 2025-12-04T09:42:12.8048633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8048707Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8048956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8049019Z return func(*args, **kwargs) 2025-12-04T09:42:12.8049258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8049371Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8049603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8049733Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8049965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.8050060Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.8050063Z 2025-12-04T09:42:12.8050170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8050365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8050436Z res = mod(**inputs) 2025-12-04T09:42:12.8050673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8050742Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8050982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8051051Z layer_outputs = layer_module( 2025-12-04T09:42:12.8051266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8051429Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8051669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8051745Z return func(*args, **kwargs) 2025-12-04T09:42:12.8051985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8052082Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8052353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8052479Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8052766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.8052851Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.8052855Z 2025-12-04T09:42:12.8052966Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8053187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8053255Z res = mod(**inputs) 2025-12-04T09:42:12.8053555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8053637Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8053873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8053951Z layer_outputs = layer_module( 2025-12-04T09:42:12.8054184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8054275Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8054518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8054582Z return func(*args, **kwargs) 2025-12-04T09:42:12.8054820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8054917Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8055155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8055274Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8055541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.8055627Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.8055633Z 2025-12-04T09:42:12.8055740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8055959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8056033Z res = mod(**inputs) 2025-12-04T09:42:12.8056290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8056365Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8056634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8056702Z layer_outputs = layer_module( 2025-12-04T09:42:12.8056925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8057009Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8057251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8057325Z return func(*args, **kwargs) 2025-12-04T09:42:12.8057567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8057652Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8057900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8058012Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8058311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.8058393Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.8058397Z 2025-12-04T09:42:12.8058503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8058707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8058768Z res = mod(**inputs) 2025-12-04T09:42:12.8059013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8059091Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8059331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8059428Z layer_outputs = layer_module( 2025-12-04T09:42:12.8059652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8059728Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8059980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8060066Z return func(*args, **kwargs) 2025-12-04T09:42:12.8060319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8060407Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8060646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8060718Z return func(*args, **kwargs) 2025-12-04T09:42:12.8060955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.8061036Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.8061284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8061375Z return func(*args, **kwargs) 2025-12-04T09:42:12.8061623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.8061701Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.8061704Z 2025-12-04T09:42:12.8061807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8062010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8062072Z res = mod(**inputs) 2025-12-04T09:42:12.8062314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8062395Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8062637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8062712Z layer_outputs = layer_module( 2025-12-04T09:42:12.8062935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8063012Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8063263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8063332Z return func(*args, **kwargs) 2025-12-04T09:42:12.8063573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8063660Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8063905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8063977Z return func(*args, **kwargs) 2025-12-04T09:42:12.8064214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.8064300Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.8064551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8064619Z return func(*args, **kwargs) 2025-12-04T09:42:12.8064864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.8064937Z key_states = self.k(current_states) 2025-12-04T09:42:12.8064941Z 2025-12-04T09:42:12.8065045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8065248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8065332Z res = mod(**inputs) 2025-12-04T09:42:12.8065573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8065650Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8065891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8065987Z layer_outputs = layer_module( 2025-12-04T09:42:12.8066224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8066302Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8066549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8066615Z return func(*args, **kwargs) 2025-12-04T09:42:12.8066856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8066941Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8067180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8067274Z return func(*args, **kwargs) 2025-12-04T09:42:12.8067521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.8067604Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.8067855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8067921Z return func(*args, **kwargs) 2025-12-04T09:42:12.8068173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.8068248Z value_states = self.v(current_states) 2025-12-04T09:42:12.8068251Z 2025-12-04T09:42:12.8068332Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8068421Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8068523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8068724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8068794Z res = mod(**inputs) 2025-12-04T09:42:12.8069045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8069126Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8069372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8069441Z layer_outputs = layer_module( 2025-12-04T09:42:12.8069673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8069752Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8069997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8070069Z return func(*args, **kwargs) 2025-12-04T09:42:12.8070314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T09:42:12.8070400Z self_attention_outputs = self.layer[0]( 2025-12-04T09:42:12.8070648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8070714Z return func(*args, **kwargs) 2025-12-04T09:42:12.8070966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T09:42:12.8071045Z attention_output = self.SelfAttention( 2025-12-04T09:42:12.8071312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8071379Z return func(*args, **kwargs) 2025-12-04T09:42:12.8071625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.8071707Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.8071711Z 2025-12-04T09:42:12.8071812Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8072023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8072109Z res = mod(**inputs) 2025-12-04T09:42:12.8072476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8072561Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8072805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8072879Z layer_outputs = layer_module( 2025-12-04T09:42:12.8073108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8073186Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8073486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8073562Z return func(*args, **kwargs) 2025-12-04T09:42:12.8073807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8073897Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8074138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8074204Z return func(*args, **kwargs) 2025-12-04T09:42:12.8074453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8074535Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8074786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8074853Z return func(*args, **kwargs) 2025-12-04T09:42:12.8075093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T09:42:12.8075180Z query_states = self.q(hidden_states) 2025-12-04T09:42:12.8075184Z 2025-12-04T09:42:12.8075287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8075484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8075555Z res = mod(**inputs) 2025-12-04T09:42:12.8075798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8075878Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8076120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8076190Z layer_outputs = layer_module( 2025-12-04T09:42:12.8076421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8076498Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8076739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8076816Z return func(*args, **kwargs) 2025-12-04T09:42:12.8077054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8077139Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8077406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8077475Z return func(*args, **kwargs) 2025-12-04T09:42:12.8077721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8077807Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8078077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8078167Z return func(*args, **kwargs) 2025-12-04T09:42:12.8078402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T09:42:12.8078484Z key_states = self.k(current_states) 2025-12-04T09:42:12.8078487Z 2025-12-04T09:42:12.8078590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8078786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8078860Z res = mod(**inputs) 2025-12-04T09:42:12.8079103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8079181Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8079441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8079511Z layer_outputs = layer_module( 2025-12-04T09:42:12.8079739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8079816Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8080056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8080128Z return func(*args, **kwargs) 2025-12-04T09:42:12.8080368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8080455Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8080704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8080771Z return func(*args, **kwargs) 2025-12-04T09:42:12.8081018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8081101Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8081347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8081412Z return func(*args, **kwargs) 2025-12-04T09:42:12.8081648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T09:42:12.8081730Z value_states = self.v(current_states) 2025-12-04T09:42:12.8081734Z 2025-12-04T09:42:12.8081814Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8081890Z cudagraph partition due to non gpu ops 2025-12-04T09:42:12.8082004Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8082210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8082284Z res = mod(**inputs) 2025-12-04T09:42:12.8082543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8082618Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8082878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8082950Z layer_outputs = layer_module( 2025-12-04T09:42:12.8083189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8083272Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8083526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8083601Z return func(*args, **kwargs) 2025-12-04T09:42:12.8083844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8083923Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8084186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8084269Z return func(*args, **kwargs) 2025-12-04T09:42:12.8084525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T09:42:12.8084618Z attention_output = self.EncDecAttention( 2025-12-04T09:42:12.8084876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8084953Z return func(*args, **kwargs) 2025-12-04T09:42:12.8085208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T09:42:12.8085305Z attn_output = self.o(attn_output) 2025-12-04T09:42:12.8085309Z 2025-12-04T09:42:12.8085426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8085634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8085708Z res = mod(**inputs) 2025-12-04T09:42:12.8085964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8086037Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8086306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8086376Z layer_outputs = layer_module( 2025-12-04T09:42:12.8086612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8086701Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8086961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8087036Z return func(*args, **kwargs) 2025-12-04T09:42:12.8087291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T09:42:12.8087375Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:42:12.8087637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8087707Z return func(*args, **kwargs) 2025-12-04T09:42:12.8087962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 529, in forward 2025-12-04T09:42:12.8088108Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:42:12.8088112Z 2025-12-04T09:42:12.8088219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8088432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8088501Z res = mod(**inputs) 2025-12-04T09:42:12.8088759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8088845Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8089103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8089182Z layer_outputs = layer_module( 2025-12-04T09:42:12.8089419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8089520Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8089786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8089855Z return func(*args, **kwargs) 2025-12-04T09:42:12.8090106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8090227Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8090497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8090628Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8090879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T09:42:12.8090985Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T09:42:12.8090989Z 2025-12-04T09:42:12.8091111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8091397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8091485Z res = mod(**inputs) 2025-12-04T09:42:12.8091778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8091858Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8092138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8092215Z layer_outputs = layer_module( 2025-12-04T09:42:12.8092465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8092568Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8092846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8092928Z return func(*args, **kwargs) 2025-12-04T09:42:12.8093183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8093278Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8093544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8093666Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8093928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T09:42:12.8094011Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T09:42:12.8094015Z 2025-12-04T09:42:12.8094123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8094335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8094403Z res = mod(**inputs) 2025-12-04T09:42:12.8094660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8094742Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8094996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8095079Z layer_outputs = layer_module( 2025-12-04T09:42:12.8095311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8095390Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8095667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8095736Z return func(*args, **kwargs) 2025-12-04T09:42:12.8096031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8096132Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8096384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8096510Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8096800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T09:42:12.8096915Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T09:42:12.8096919Z 2025-12-04T09:42:12.8097036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8097241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8097315Z res = mod(**inputs) 2025-12-04T09:42:12.8097582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T09:42:12.8097658Z decoder_outputs = self.decoder( 2025-12-04T09:42:12.8097916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T09:42:12.8098009Z layer_outputs = layer_module( 2025-12-04T09:42:12.8098244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:12.8098335Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:12.8098592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:12.8098672Z return func(*args, **kwargs) 2025-12-04T09:42:12.8098943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T09:42:12.8099033Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:42:12.8099304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T09:42:12.8099423Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:42:12.8099700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T09:42:12.8099786Z hidden_states = self.wo(hidden_states) 2025-12-04T09:42:12.8099790Z 2025-12-04T09:42:12.8099904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8100117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8100182Z res = mod(**inputs) 2025-12-04T09:42:12.8100496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1816, in forward 2025-12-04T09:42:12.8100594Z lm_logits = self.lm_head(sequence_output) 2025-12-04T09:42:12.8100598Z 2025-12-04T09:42:12.8100704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:12.8100930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:12.8101005Z res = mod(**inputs) 2025-12-04T09:42:12.8101237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-12-04T09:42:12.8101384Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-12-04T09:42:12.8101390Z 2025-12-04T09:42:23.1465328Z Compilation time (from dynamo_timed): 21.251319495 2025-12-04T09:42:23.1639775Z pass 2025-12-04T09:42:23.1642411Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:23.1643372Z TIMING: _recursive_pre_grad_passes:0.01409 _recursive_joint_graph_passes:0.78402 _recursive_post_grad_passes:0.08889 async_compile.wait:0.73973 code_gen:9.94162 inductor_compile:12.2088 backend_compile:17.34595 gc:0.00019 entire_frame_compile:21.25132 total_wall_time:21.25132 2025-12-04T09:42:23.1644890Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:16428 | FakeTensor.__torch_dispatch__:6387 | ProxyTorchDispatchMode.__torch_dispatch__:4865 2025-12-04T09:42:23.1645414Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-12-04T09:42:25.6591225Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:42:25.6592615Z import pynvml # type: ignore[import] 2025-12-04T09:42:28.9806428Z 2025-12-04T09:42:28.9934411Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T09:42:28.9937677Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T09:42:31.6646752Z 2025-12-04T09:42:31.6647568Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:42:31.6677109Z cpu eval MegatronBertForCausalLM 2025-12-04T09:42:33.2385203Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:33.8475263Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:34.4572065Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:42:48.6812249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6813113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6813777Z res = mod(**inputs) 2025-12-04T09:42:48.6814595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6815493Z outputs = self.bert( 2025-12-04T09:42:48.6816104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6816621Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6817407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6818188Z layer_outputs = layer_module( 2025-12-04T09:42:48.6818861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6819583Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6820291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6820996Z return func(*args, **kwargs) 2025-12-04T09:42:48.6821758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.6822575Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.6823282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6823955Z return func(*args, **kwargs) 2025-12-04T09:42:48.6824705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.6825508Z self_outputs = self.self( 2025-12-04T09:42:48.6826184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6826873Z return func(*args, **kwargs) 2025-12-04T09:42:48.6827636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.6828807Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.6829075Z 2025-12-04T09:42:48.6829258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6829904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6830490Z res = mod(**inputs) 2025-12-04T09:42:48.6831341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6832165Z outputs = self.bert( 2025-12-04T09:42:48.6832860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6833635Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6834358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6835104Z layer_outputs = layer_module( 2025-12-04T09:42:48.6835749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6836395Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6837152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6837800Z return func(*args, **kwargs) 2025-12-04T09:42:48.6838528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.6839322Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.6840006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6840679Z return func(*args, **kwargs) 2025-12-04T09:42:48.6841403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.6842182Z self_outputs = self.self( 2025-12-04T09:42:48.6842821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6843525Z return func(*args, **kwargs) 2025-12-04T09:42:48.6844264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.6845038Z key_layer = self.key(current_states) 2025-12-04T09:42:48.6845278Z 2025-12-04T09:42:48.6845467Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6846078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6846651Z res = mod(**inputs) 2025-12-04T09:42:48.6847374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6848149Z outputs = self.bert( 2025-12-04T09:42:48.6848861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6849624Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6850381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6851203Z layer_outputs = layer_module( 2025-12-04T09:42:48.6851960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6852656Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6853346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6854025Z return func(*args, **kwargs) 2025-12-04T09:42:48.6854857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.6855658Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.6856360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6857026Z return func(*args, **kwargs) 2025-12-04T09:42:48.6857823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.6858614Z self_outputs = self.self( 2025-12-04T09:42:48.6859281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6859952Z return func(*args, **kwargs) 2025-12-04T09:42:48.6860700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.6861502Z value_layer = self.value(current_states) 2025-12-04T09:42:48.6861731Z 2025-12-04T09:42:48.6861869Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.6862225Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.6862685Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6863328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6863878Z res = mod(**inputs) 2025-12-04T09:42:48.6864613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6865350Z outputs = self.bert( 2025-12-04T09:42:48.6866052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6866844Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6867629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6868397Z layer_outputs = layer_module( 2025-12-04T09:42:48.6869008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6869640Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6870335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6871007Z return func(*args, **kwargs) 2025-12-04T09:42:48.6871776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.6872794Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.6873529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6874199Z return func(*args, **kwargs) 2025-12-04T09:42:48.6874942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.6875788Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.6876616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.6877407Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.6877673Z 2025-12-04T09:42:48.6877832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6878481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6879064Z res = mod(**inputs) 2025-12-04T09:42:48.6879868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6880647Z outputs = self.bert( 2025-12-04T09:42:48.6881348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6882132Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6882961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6883775Z layer_outputs = layer_module( 2025-12-04T09:42:48.6884373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6885005Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6885678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6886369Z return func(*args, **kwargs) 2025-12-04T09:42:48.6887159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.6888051Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.6888811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.6889586Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.6890439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.6891431Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.6892296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.6893108Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.6893370Z 2025-12-04T09:42:48.6893552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6894201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6894753Z res = mod(**inputs) 2025-12-04T09:42:48.6895489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6896265Z outputs = self.bert( 2025-12-04T09:42:48.6896982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6897810Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6898633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6899471Z layer_outputs = layer_module( 2025-12-04T09:42:48.6900099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6900787Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6901472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6902161Z return func(*args, **kwargs) 2025-12-04T09:42:48.6902909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.6903689Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.6904426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.6905149Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.6906020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.6906993Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.6907818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.6908410Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.6908830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.6909279Z return self.act(input) 2025-12-04T09:42:48.6909467Z 2025-12-04T09:42:48.6909636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6910243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6910818Z res = mod(**inputs) 2025-12-04T09:42:48.6911501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6912248Z outputs = self.bert( 2025-12-04T09:42:48.6912983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6913761Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6914595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6915368Z layer_outputs = layer_module( 2025-12-04T09:42:48.6915829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6916221Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6916725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6917407Z return func(*args, **kwargs) 2025-12-04T09:42:48.6918153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.6918940Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.6919637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.6920346Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.6921195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.6922182Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.6922989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.6923700Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.6923920Z 2025-12-04T09:42:48.6924072Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6924662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6925173Z res = mod(**inputs) 2025-12-04T09:42:48.6944638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6945599Z outputs = self.bert( 2025-12-04T09:42:48.6946274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6947007Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6947749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6948449Z layer_outputs = layer_module( 2025-12-04T09:42:48.6948991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6949769Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6950447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6951123Z return func(*args, **kwargs) 2025-12-04T09:42:48.6951862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.6952630Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.6953303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6953871Z return func(*args, **kwargs) 2025-12-04T09:42:48.6954526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.6955212Z self_outputs = self.self( 2025-12-04T09:42:48.6955826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6956445Z return func(*args, **kwargs) 2025-12-04T09:42:48.6957128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.6957928Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.6958149Z 2025-12-04T09:42:48.6958323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6958930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6959471Z res = mod(**inputs) 2025-12-04T09:42:48.6960167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6960914Z outputs = self.bert( 2025-12-04T09:42:48.6961593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6962308Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6963018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6963711Z layer_outputs = layer_module( 2025-12-04T09:42:48.6964334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6964993Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6965692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6967100Z return func(*args, **kwargs) 2025-12-04T09:42:48.6967898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.6968700Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.6969368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6970069Z return func(*args, **kwargs) 2025-12-04T09:42:48.6970841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.6971828Z self_outputs = self.self( 2025-12-04T09:42:48.6972861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6973568Z return func(*args, **kwargs) 2025-12-04T09:42:48.6974334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.6975171Z key_layer = self.key(current_states) 2025-12-04T09:42:48.6975436Z 2025-12-04T09:42:48.6976970Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6977696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6978297Z res = mod(**inputs) 2025-12-04T09:42:48.6979036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6979868Z outputs = self.bert( 2025-12-04T09:42:48.6980765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6981691Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.6982499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.6983354Z layer_outputs = layer_module( 2025-12-04T09:42:48.6984062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.6984775Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.6985529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6986300Z return func(*args, **kwargs) 2025-12-04T09:42:48.6987117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.6987994Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.6988778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6989489Z return func(*args, **kwargs) 2025-12-04T09:42:48.6990292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.6991058Z self_outputs = self.self( 2025-12-04T09:42:48.6991762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.6992482Z return func(*args, **kwargs) 2025-12-04T09:42:48.6993250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.6994091Z value_layer = self.value(current_states) 2025-12-04T09:42:48.6994357Z 2025-12-04T09:42:48.6994497Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.6994865Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.6995300Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.6995922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.6996415Z res = mod(**inputs) 2025-12-04T09:42:48.6997139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.6997905Z outputs = self.bert( 2025-12-04T09:42:48.6998638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.6999419Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7000194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7000977Z layer_outputs = layer_module( 2025-12-04T09:42:48.7001625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7002301Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7002999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7003699Z return func(*args, **kwargs) 2025-12-04T09:42:48.7004555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7005363Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7006075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7006765Z return func(*args, **kwargs) 2025-12-04T09:42:48.7007579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7008494Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7009357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7010152Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7010396Z 2025-12-04T09:42:48.7010588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7011408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7012055Z res = mod(**inputs) 2025-12-04T09:42:48.7012892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7013853Z outputs = self.bert( 2025-12-04T09:42:48.7014673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7015545Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7016421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7017314Z layer_outputs = layer_module( 2025-12-04T09:42:48.7018012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7018767Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7019546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7020269Z return func(*args, **kwargs) 2025-12-04T09:42:48.7021038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7021864Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7022624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7023352Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7024183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7025110Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7025974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7026791Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7027059Z 2025-12-04T09:42:48.7027243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7027934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7028524Z res = mod(**inputs) 2025-12-04T09:42:48.7029274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7030084Z outputs = self.bert( 2025-12-04T09:42:48.7030826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7031628Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7032541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7033347Z layer_outputs = layer_module( 2025-12-04T09:42:48.7034022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7034731Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7035469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7036176Z return func(*args, **kwargs) 2025-12-04T09:42:48.7036921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7037728Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7038491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7039222Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7040057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7041017Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7041866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7042737Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7043417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7044057Z return self.act(input) 2025-12-04T09:42:48.7044243Z 2025-12-04T09:42:48.7044438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7045068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7045645Z res = mod(**inputs) 2025-12-04T09:42:48.7046388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7047160Z outputs = self.bert( 2025-12-04T09:42:48.7047904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7048728Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7049542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7050348Z layer_outputs = layer_module( 2025-12-04T09:42:48.7050995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7051850Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7052648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7053358Z return func(*args, **kwargs) 2025-12-04T09:42:48.7054123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7054949Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7055692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7056457Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7057298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7058262Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7059237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7060039Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7060284Z 2025-12-04T09:42:48.7060456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7061154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7061735Z res = mod(**inputs) 2025-12-04T09:42:48.7062540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7063318Z outputs = self.bert( 2025-12-04T09:42:48.7064071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7064870Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7065665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7066452Z layer_outputs = layer_module( 2025-12-04T09:42:48.7067100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7067787Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7068492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7069188Z return func(*args, **kwargs) 2025-12-04T09:42:48.7069942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7070747Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7071503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7072439Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7073272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7074233Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7075136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.7075903Z return input_tensor + hidden_states 2025-12-04T09:42:48.7076136Z 2025-12-04T09:42:48.7076318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7076992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7077576Z res = mod(**inputs) 2025-12-04T09:42:48.7078309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7079092Z outputs = self.bert( 2025-12-04T09:42:48.7079821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7080610Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7081409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7082187Z layer_outputs = layer_module( 2025-12-04T09:42:48.7082821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7083469Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7084153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7084836Z return func(*args, **kwargs) 2025-12-04T09:42:48.7085750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7086558Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7087304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7088014Z return func(*args, **kwargs) 2025-12-04T09:42:48.7088850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7089669Z self_outputs = self.self( 2025-12-04T09:42:48.7090347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7091062Z return func(*args, **kwargs) 2025-12-04T09:42:48.7092015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7092885Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7093137Z 2025-12-04T09:42:48.7093321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7094057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7094655Z res = mod(**inputs) 2025-12-04T09:42:48.7095413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7096215Z outputs = self.bert( 2025-12-04T09:42:48.7096966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7097753Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7098572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7099385Z layer_outputs = layer_module( 2025-12-04T09:42:48.7100032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7100706Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7101433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7102148Z return func(*args, **kwargs) 2025-12-04T09:42:48.7102939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7103776Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7104502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7105185Z return func(*args, **kwargs) 2025-12-04T09:42:48.7105937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7106713Z self_outputs = self.self( 2025-12-04T09:42:48.7107364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7108030Z return func(*args, **kwargs) 2025-12-04T09:42:48.7108798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7109581Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7109815Z 2025-12-04T09:42:48.7109997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7110646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7111262Z res = mod(**inputs) 2025-12-04T09:42:48.7112056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7112842Z outputs = self.bert( 2025-12-04T09:42:48.7113573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7114331Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7115105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7115876Z layer_outputs = layer_module( 2025-12-04T09:42:48.7116485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7117118Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7117773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7118424Z return func(*args, **kwargs) 2025-12-04T09:42:48.7119159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7119928Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7120657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7121317Z return func(*args, **kwargs) 2025-12-04T09:42:48.7122056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7122839Z self_outputs = self.self( 2025-12-04T09:42:48.7123471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7124155Z return func(*args, **kwargs) 2025-12-04T09:42:48.7124895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7125688Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7125934Z 2025-12-04T09:42:48.7126063Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7126500Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7126892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7127515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7128073Z res = mod(**inputs) 2025-12-04T09:42:48.7128795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7129564Z outputs = self.bert( 2025-12-04T09:42:48.7130283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7131061Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7131952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7132739Z layer_outputs = layer_module( 2025-12-04T09:42:48.7133403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7134082Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7134788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7135471Z return func(*args, **kwargs) 2025-12-04T09:42:48.7136199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7136997Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7137742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7138417Z return func(*args, **kwargs) 2025-12-04T09:42:48.7139143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7140000Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7140874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7141669Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7141904Z 2025-12-04T09:42:48.7142078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7142721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7143289Z res = mod(**inputs) 2025-12-04T09:42:48.7144007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7144759Z outputs = self.bert( 2025-12-04T09:42:48.7145482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7146279Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7147050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7147820Z layer_outputs = layer_module( 2025-12-04T09:42:48.7148431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7149038Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7149695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7150356Z return func(*args, **kwargs) 2025-12-04T09:42:48.7151058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7151793Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7152507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7153222Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7154031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7154889Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7155676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7156429Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7156671Z 2025-12-04T09:42:48.7156848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7157464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7158018Z res = mod(**inputs) 2025-12-04T09:42:48.7158715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7159454Z outputs = self.bert( 2025-12-04T09:42:48.7160161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7160965Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7161734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7162489Z layer_outputs = layer_module( 2025-12-04T09:42:48.7163153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7163794Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7164492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7165157Z return func(*args, **kwargs) 2025-12-04T09:42:48.7165958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7166773Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7167506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7168216Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7169018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7169894Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7170709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7171710Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7172610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7173298Z return self.act(input) 2025-12-04T09:42:48.7173505Z 2025-12-04T09:42:48.7173698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7174362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7174947Z res = mod(**inputs) 2025-12-04T09:42:48.7175686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7176469Z outputs = self.bert( 2025-12-04T09:42:48.7177211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7178025Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7178788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7179572Z layer_outputs = layer_module( 2025-12-04T09:42:48.7180184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7180840Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7181540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7182217Z return func(*args, **kwargs) 2025-12-04T09:42:48.7182959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7183742Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7184448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7185148Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7185979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7186907Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7187785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7188581Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7188836Z 2025-12-04T09:42:48.7189118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7189758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7190322Z res = mod(**inputs) 2025-12-04T09:42:48.7191043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7191788Z outputs = self.bert( 2025-12-04T09:42:48.7192612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7193423Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7194186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7194956Z layer_outputs = layer_module( 2025-12-04T09:42:48.7195597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7196247Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7196937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7197655Z return func(*args, **kwargs) 2025-12-04T09:42:48.7198367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7199131Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7199816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7200470Z return func(*args, **kwargs) 2025-12-04T09:42:48.7201173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7201910Z self_outputs = self.self( 2025-12-04T09:42:48.7202521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7203161Z return func(*args, **kwargs) 2025-12-04T09:42:48.7203873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7204649Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7204879Z 2025-12-04T09:42:48.7205061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7205687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7206251Z res = mod(**inputs) 2025-12-04T09:42:48.7206960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7207729Z outputs = self.bert( 2025-12-04T09:42:48.7208447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7209217Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7209993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7210754Z layer_outputs = layer_module( 2025-12-04T09:42:48.7211478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7212158Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7212876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7213568Z return func(*args, **kwargs) 2025-12-04T09:42:48.7214337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7215233Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7215974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7216691Z return func(*args, **kwargs) 2025-12-04T09:42:48.7217675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7218726Z self_outputs = self.self( 2025-12-04T09:42:48.7219566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7220451Z return func(*args, **kwargs) 2025-12-04T09:42:48.7221355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7222029Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7222241Z 2025-12-04T09:42:48.7222412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7222985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7223503Z res = mod(**inputs) 2025-12-04T09:42:48.7224206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7224888Z outputs = self.bert( 2025-12-04T09:42:48.7225539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7226288Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7227187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7227903Z layer_outputs = layer_module( 2025-12-04T09:42:48.7228492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7229138Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7229792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7230453Z return func(*args, **kwargs) 2025-12-04T09:42:48.7231162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7231950Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7232638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7233293Z return func(*args, **kwargs) 2025-12-04T09:42:48.7234005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7234761Z self_outputs = self.self( 2025-12-04T09:42:48.7235429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7236132Z return func(*args, **kwargs) 2025-12-04T09:42:48.7236902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7237705Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7237969Z 2025-12-04T09:42:48.7238102Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7238465Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7238865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7239506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7240087Z res = mod(**inputs) 2025-12-04T09:42:48.7240895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7241656Z outputs = self.bert( 2025-12-04T09:42:48.7242398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7243187Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7244000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7244810Z layer_outputs = layer_module( 2025-12-04T09:42:48.7245448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7246129Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7246843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7247528Z return func(*args, **kwargs) 2025-12-04T09:42:48.7248306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7249134Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7249901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7250606Z return func(*args, **kwargs) 2025-12-04T09:42:48.7251528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7252511Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7253483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7254323Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7254585Z 2025-12-04T09:42:48.7254794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7255487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7256087Z res = mod(**inputs) 2025-12-04T09:42:48.7256866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7257678Z outputs = self.bert( 2025-12-04T09:42:48.7258426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7259233Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7260039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7260843Z layer_outputs = layer_module( 2025-12-04T09:42:48.7261496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7262167Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7262892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7263583Z return func(*args, **kwargs) 2025-12-04T09:42:48.7264355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7265180Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7265947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7266693Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7267558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7268559Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7269390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7270184Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7270431Z 2025-12-04T09:42:48.7270637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7271289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7271882Z res = mod(**inputs) 2025-12-04T09:42:48.7272745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7273536Z outputs = self.bert( 2025-12-04T09:42:48.7274273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7275060Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7275817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7276662Z layer_outputs = layer_module( 2025-12-04T09:42:48.7277290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7277954Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7278640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7279308Z return func(*args, **kwargs) 2025-12-04T09:42:48.7280055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7280847Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7281578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7282296Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7283104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7283979Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7284840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7285661Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7286344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7286963Z return self.act(input) 2025-12-04T09:42:48.7287147Z 2025-12-04T09:42:48.7287330Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7287974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7288577Z res = mod(**inputs) 2025-12-04T09:42:48.7289337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7290130Z outputs = self.bert( 2025-12-04T09:42:48.7290859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7291880Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7292705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7293493Z layer_outputs = layer_module( 2025-12-04T09:42:48.7294126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7294904Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7295631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7296315Z return func(*args, **kwargs) 2025-12-04T09:42:48.7297144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7298027Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7298773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7299521Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7300361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7301329Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7302224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7303041Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7303330Z 2025-12-04T09:42:48.7303506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7304167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7304732Z res = mod(**inputs) 2025-12-04T09:42:48.7305473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7306263Z outputs = self.bert( 2025-12-04T09:42:48.7306996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7307782Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7308558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7309344Z layer_outputs = layer_module( 2025-12-04T09:42:48.7309994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7310682Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7311400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7312106Z return func(*args, **kwargs) 2025-12-04T09:42:48.7312890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7313723Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7314492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7315244Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7316099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7317094Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7318002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.7318815Z return input_tensor + hidden_states 2025-12-04T09:42:48.7319048Z 2025-12-04T09:42:48.7319236Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7319919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7320499Z res = mod(**inputs) 2025-12-04T09:42:48.7321313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7322145Z outputs = self.bert( 2025-12-04T09:42:48.7322902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7323713Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7324557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7325403Z layer_outputs = layer_module( 2025-12-04T09:42:48.7326060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7326746Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7327475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7328185Z return func(*args, **kwargs) 2025-12-04T09:42:48.7328984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7329815Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7330586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7331412Z return func(*args, **kwargs) 2025-12-04T09:42:48.7332277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7333128Z self_outputs = self.self( 2025-12-04T09:42:48.7333856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7334608Z return func(*args, **kwargs) 2025-12-04T09:42:48.7335375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7336242Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7336495Z 2025-12-04T09:42:48.7336694Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7337394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7337991Z res = mod(**inputs) 2025-12-04T09:42:48.7338768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7339580Z outputs = self.bert( 2025-12-04T09:42:48.7340323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7341108Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7341882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7342682Z layer_outputs = layer_module( 2025-12-04T09:42:48.7343324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7344004Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7344709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7345402Z return func(*args, **kwargs) 2025-12-04T09:42:48.7346159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7346978Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7347680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7348364Z return func(*args, **kwargs) 2025-12-04T09:42:48.7349192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7349982Z self_outputs = self.self( 2025-12-04T09:42:48.7350636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7351343Z return func(*args, **kwargs) 2025-12-04T09:42:48.7352092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7352902Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7353141Z 2025-12-04T09:42:48.7353316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7353974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7354568Z res = mod(**inputs) 2025-12-04T09:42:48.7355335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7356129Z outputs = self.bert( 2025-12-04T09:42:48.7356886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7357712Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7358486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7359266Z layer_outputs = layer_module( 2025-12-04T09:42:48.7359891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7360541Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7361223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7361924Z return func(*args, **kwargs) 2025-12-04T09:42:48.7362731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7363594Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7364322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7365012Z return func(*args, **kwargs) 2025-12-04T09:42:48.7365779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7366560Z self_outputs = self.self( 2025-12-04T09:42:48.7367212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7367901Z return func(*args, **kwargs) 2025-12-04T09:42:48.7368656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7369454Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7369694Z 2025-12-04T09:42:48.7369826Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7370181Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7370567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7371350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7372028Z res = mod(**inputs) 2025-12-04T09:42:48.7373018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7373893Z outputs = self.bert( 2025-12-04T09:42:48.7374706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7375701Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7376585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7377454Z layer_outputs = layer_module( 2025-12-04T09:42:48.7378230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7378981Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7379801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7380556Z return func(*args, **kwargs) 2025-12-04T09:42:48.7381317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7382141Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7382847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7383524Z return func(*args, **kwargs) 2025-12-04T09:42:48.7384274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7385293Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7386170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7386983Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7387219Z 2025-12-04T09:42:48.7387404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7388029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7388611Z res = mod(**inputs) 2025-12-04T09:42:48.7389349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7390168Z outputs = self.bert( 2025-12-04T09:42:48.7390898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7391688Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7392475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7393266Z layer_outputs = layer_module( 2025-12-04T09:42:48.7393897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7394564Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7395279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7395935Z return func(*args, **kwargs) 2025-12-04T09:42:48.7396679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7397464Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7398212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7398913Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7399728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7400611Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7401433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7402259Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7402512Z 2025-12-04T09:42:48.7402684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7403326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7403898Z res = mod(**inputs) 2025-12-04T09:42:48.7404657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7405478Z outputs = self.bert( 2025-12-04T09:42:48.7406209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7406992Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7407768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7408576Z layer_outputs = layer_module( 2025-12-04T09:42:48.7409210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7409871Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7410595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7411396Z return func(*args, **kwargs) 2025-12-04T09:42:48.7412246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7413106Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7413849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7414585Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7415439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7416346Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7417193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7418057Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7418772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7419399Z return self.act(input) 2025-12-04T09:42:48.7419590Z 2025-12-04T09:42:48.7419778Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7420441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7421041Z res = mod(**inputs) 2025-12-04T09:42:48.7421784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7422566Z outputs = self.bert( 2025-12-04T09:42:48.7423302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7424106Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7424909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7425707Z layer_outputs = layer_module( 2025-12-04T09:42:48.7426368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7427048Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7427759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7428457Z return func(*args, **kwargs) 2025-12-04T09:42:48.7429283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7430109Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7430857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7431610Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7432481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7433430Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7434339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7435152Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7435414Z 2025-12-04T09:42:48.7435589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7436253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7436890Z res = mod(**inputs) 2025-12-04T09:42:48.7437614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7438379Z outputs = self.bert( 2025-12-04T09:42:48.7439102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7439867Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7440602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7441369Z layer_outputs = layer_module( 2025-12-04T09:42:48.7441991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7442642Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7443331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7443996Z return func(*args, **kwargs) 2025-12-04T09:42:48.7444748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7445528Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7446224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7446882Z return func(*args, **kwargs) 2025-12-04T09:42:48.7447604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7448373Z self_outputs = self.self( 2025-12-04T09:42:48.7449030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7449689Z return func(*args, **kwargs) 2025-12-04T09:42:48.7450424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7451351Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7451602Z 2025-12-04T09:42:48.7451795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7452432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7453021Z res = mod(**inputs) 2025-12-04T09:42:48.7453763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7454591Z outputs = self.bert( 2025-12-04T09:42:48.7455332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7456135Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7456919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7457740Z layer_outputs = layer_module( 2025-12-04T09:42:48.7458438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7459097Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7459793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7460479Z return func(*args, **kwargs) 2025-12-04T09:42:48.7461242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7462041Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7462731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7463441Z return func(*args, **kwargs) 2025-12-04T09:42:48.7464216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7465010Z self_outputs = self.self( 2025-12-04T09:42:48.7465649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7466308Z return func(*args, **kwargs) 2025-12-04T09:42:48.7467023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7467791Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7468018Z 2025-12-04T09:42:48.7468190Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7468834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7469414Z res = mod(**inputs) 2025-12-04T09:42:48.7470157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7470931Z outputs = self.bert( 2025-12-04T09:42:48.7471647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7472677Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7473425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7474159Z layer_outputs = layer_module( 2025-12-04T09:42:48.7474764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7475402Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7476078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7476755Z return func(*args, **kwargs) 2025-12-04T09:42:48.7477492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7478244Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7478925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7479569Z return func(*args, **kwargs) 2025-12-04T09:42:48.7480389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7481137Z self_outputs = self.self( 2025-12-04T09:42:48.7481760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7482404Z return func(*args, **kwargs) 2025-12-04T09:42:48.7483175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7483995Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7484247Z 2025-12-04T09:42:48.7484376Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7484735Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7485123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7485762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7486332Z res = mod(**inputs) 2025-12-04T09:42:48.7487047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7487824Z outputs = self.bert( 2025-12-04T09:42:48.7488554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7489378Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7490144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7490918Z layer_outputs = layer_module( 2025-12-04T09:42:48.7491685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7492411Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7493147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7493822Z return func(*args, **kwargs) 2025-12-04T09:42:48.7494568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7495332Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7496015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7496689Z return func(*args, **kwargs) 2025-12-04T09:42:48.7497422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7498273Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7499121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7499914Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7500154Z 2025-12-04T09:42:48.7500337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7500972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7501547Z res = mod(**inputs) 2025-12-04T09:42:48.7502278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7503064Z outputs = self.bert( 2025-12-04T09:42:48.7503773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7504554Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7505292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7506046Z layer_outputs = layer_module( 2025-12-04T09:42:48.7506735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7507394Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7508062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7508713Z return func(*args, **kwargs) 2025-12-04T09:42:48.7509458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7510320Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7511038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7511758Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7512579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7513467Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7514295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7515130Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7515371Z 2025-12-04T09:42:48.7515555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7516193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7516760Z res = mod(**inputs) 2025-12-04T09:42:48.7517464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7518213Z outputs = self.bert( 2025-12-04T09:42:48.7518888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7519631Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7520368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7521114Z layer_outputs = layer_module( 2025-12-04T09:42:48.7521721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7522355Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7523027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7523690Z return func(*args, **kwargs) 2025-12-04T09:42:48.7524424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7525196Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7525931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7526661Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7527487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7528381Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7529202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7530017Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7530698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7531467Z return self.act(input) 2025-12-04T09:42:48.7531672Z 2025-12-04T09:42:48.7531938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7532630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7533214Z res = mod(**inputs) 2025-12-04T09:42:48.7533935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7534740Z outputs = self.bert( 2025-12-04T09:42:48.7535499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7536275Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7537031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7537787Z layer_outputs = layer_module( 2025-12-04T09:42:48.7538427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7539085Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7539772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7540490Z return func(*args, **kwargs) 2025-12-04T09:42:48.7541240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7542046Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7542771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7543491Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7544305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7545250Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7546127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7546919Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7547164Z 2025-12-04T09:42:48.7547351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7547994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7548554Z res = mod(**inputs) 2025-12-04T09:42:48.7549285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7550062Z outputs = self.bert( 2025-12-04T09:42:48.7550767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7551524Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7552245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7552987Z layer_outputs = layer_module( 2025-12-04T09:42:48.7553598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7554246Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7554923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7555592Z return func(*args, **kwargs) 2025-12-04T09:42:48.7556312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7557079Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7557896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7558593Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7559408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7560398Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7561304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.7562075Z return input_tensor + hidden_states 2025-12-04T09:42:48.7562312Z 2025-12-04T09:42:48.7562484Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7563120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7563678Z res = mod(**inputs) 2025-12-04T09:42:48.7564367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7565128Z outputs = self.bert( 2025-12-04T09:42:48.7565847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7566637Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7567397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7568192Z layer_outputs = layer_module( 2025-12-04T09:42:48.7568841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7569500Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7570250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7570939Z return func(*args, **kwargs) 2025-12-04T09:42:48.7571797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7572857Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7573584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7574287Z return func(*args, **kwargs) 2025-12-04T09:42:48.7575051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7575855Z self_outputs = self.self( 2025-12-04T09:42:48.7576511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7577180Z return func(*args, **kwargs) 2025-12-04T09:42:48.7577942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7578737Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7578979Z 2025-12-04T09:42:48.7579167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7579831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7580410Z res = mod(**inputs) 2025-12-04T09:42:48.7581138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7581914Z outputs = self.bert( 2025-12-04T09:42:48.7582630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7583406Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7584294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7585096Z layer_outputs = layer_module( 2025-12-04T09:42:48.7585730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7586400Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7587167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7587896Z return func(*args, **kwargs) 2025-12-04T09:42:48.7588674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7589475Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7590197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7590877Z return func(*args, **kwargs) 2025-12-04T09:42:48.7591639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7592447Z self_outputs = self.self( 2025-12-04T09:42:48.7593103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7593776Z return func(*args, **kwargs) 2025-12-04T09:42:48.7594513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7595285Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7595521Z 2025-12-04T09:42:48.7595699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7596361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7596949Z res = mod(**inputs) 2025-12-04T09:42:48.7597706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7598464Z outputs = self.bert( 2025-12-04T09:42:48.7599176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7599949Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7600735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7601506Z layer_outputs = layer_module( 2025-12-04T09:42:48.7602137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7602807Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7603510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7604214Z return func(*args, **kwargs) 2025-12-04T09:42:48.7604987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7605812Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7606545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7607247Z return func(*args, **kwargs) 2025-12-04T09:42:48.7608034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7608826Z self_outputs = self.self( 2025-12-04T09:42:48.7609496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7610324Z return func(*args, **kwargs) 2025-12-04T09:42:48.7611151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7616525Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7616810Z 2025-12-04T09:42:48.7616957Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7617339Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7617839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7618564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7619202Z res = mod(**inputs) 2025-12-04T09:42:48.7619982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7620840Z outputs = self.bert( 2025-12-04T09:42:48.7621648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7622511Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7623355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7624336Z layer_outputs = layer_module( 2025-12-04T09:42:48.7625006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7625694Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7626409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7627120Z return func(*args, **kwargs) 2025-12-04T09:42:48.7627909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7628763Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7629498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7630207Z return func(*args, **kwargs) 2025-12-04T09:42:48.7630971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7631891Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7632792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7633611Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7633856Z 2025-12-04T09:42:48.7634032Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7634676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7635248Z res = mod(**inputs) 2025-12-04T09:42:48.7635958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7636740Z outputs = self.bert( 2025-12-04T09:42:48.7637488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7638287Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7639077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7639870Z layer_outputs = layer_module( 2025-12-04T09:42:48.7640518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7641173Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7641932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7642630Z return func(*args, **kwargs) 2025-12-04T09:42:48.7643384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7644409Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7645232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7645964Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7646792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7647697Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7648554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7649378Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7649634Z 2025-12-04T09:42:48.7649812Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7650529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7651138Z res = mod(**inputs) 2025-12-04T09:42:48.7652009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7652818Z outputs = self.bert( 2025-12-04T09:42:48.7653599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7654400Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7655181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7655978Z layer_outputs = layer_module( 2025-12-04T09:42:48.7656634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7657340Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7658056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7658773Z return func(*args, **kwargs) 2025-12-04T09:42:48.7659537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7660329Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7661091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7661837Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7662669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7663590Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7664464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7665360Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7666086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7666725Z return self.act(input) 2025-12-04T09:42:48.7666920Z 2025-12-04T09:42:48.7667101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7667775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7668350Z res = mod(**inputs) 2025-12-04T09:42:48.7669177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7670045Z outputs = self.bert( 2025-12-04T09:42:48.7670803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7671702Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7672895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7673720Z layer_outputs = layer_module( 2025-12-04T09:42:48.7674361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7675060Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7675779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7676487Z return func(*args, **kwargs) 2025-12-04T09:42:48.7677257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7678139Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7678913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7679674Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7680539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7681521Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7682463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7683304Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7683570Z 2025-12-04T09:42:48.7683754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7684417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7685024Z res = mod(**inputs) 2025-12-04T09:42:48.7685729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7686504Z outputs = self.bert( 2025-12-04T09:42:48.7687249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7688040Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7688795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7689572Z layer_outputs = layer_module( 2025-12-04T09:42:48.7690196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7690846Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7691667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7692396Z return func(*args, **kwargs) 2025-12-04T09:42:48.7693216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7694065Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7694790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7695473Z return func(*args, **kwargs) 2025-12-04T09:42:48.7696356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7697132Z self_outputs = self.self( 2025-12-04T09:42:48.7697801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7698566Z return func(*args, **kwargs) 2025-12-04T09:42:48.7699361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7700152Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7700402Z 2025-12-04T09:42:48.7700577Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7701245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7701820Z res = mod(**inputs) 2025-12-04T09:42:48.7702560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7703346Z outputs = self.bert( 2025-12-04T09:42:48.7704099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7704920Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7705709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7706504Z layer_outputs = layer_module( 2025-12-04T09:42:48.7707134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7707801Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7708505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7709168Z return func(*args, **kwargs) 2025-12-04T09:42:48.7709926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7710751Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7711476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7712162Z return func(*args, **kwargs) 2025-12-04T09:42:48.7712955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7713730Z self_outputs = self.self( 2025-12-04T09:42:48.7714402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7715090Z return func(*args, **kwargs) 2025-12-04T09:42:48.7715850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7716659Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7716890Z 2025-12-04T09:42:48.7717066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7717729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7718295Z res = mod(**inputs) 2025-12-04T09:42:48.7719030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7719809Z outputs = self.bert( 2025-12-04T09:42:48.7720533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7721316Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7722115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7722999Z layer_outputs = layer_module( 2025-12-04T09:42:48.7723669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7724400Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7725181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7725998Z return func(*args, **kwargs) 2025-12-04T09:42:48.7726781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7727675Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7728466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7729201Z return func(*args, **kwargs) 2025-12-04T09:42:48.7730033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7730848Z self_outputs = self.self( 2025-12-04T09:42:48.7731641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7732473Z return func(*args, **kwargs) 2025-12-04T09:42:48.7733348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7734170Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7734426Z 2025-12-04T09:42:48.7734571Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7734923Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7735342Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7736006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7736605Z res = mod(**inputs) 2025-12-04T09:42:48.7737369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7738174Z outputs = self.bert( 2025-12-04T09:42:48.7738938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7739754Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7740559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7741377Z layer_outputs = layer_module( 2025-12-04T09:42:48.7742027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7742713Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7743434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7744129Z return func(*args, **kwargs) 2025-12-04T09:42:48.7744905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7745737Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7746451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7747141Z return func(*args, **kwargs) 2025-12-04T09:42:48.7747914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7748825Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7749800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7750633Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7750904Z 2025-12-04T09:42:48.7751092Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7751842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7752455Z res = mod(**inputs) 2025-12-04T09:42:48.7753214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7753980Z outputs = self.bert( 2025-12-04T09:42:48.7754720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7755505Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7756253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7757019Z layer_outputs = layer_module( 2025-12-04T09:42:48.7757649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7758295Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7759045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7759712Z return func(*args, **kwargs) 2025-12-04T09:42:48.7760445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7761225Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7761955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7762676Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7763487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7764369Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7765193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7765990Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7766224Z 2025-12-04T09:42:48.7766395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7767018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7767584Z res = mod(**inputs) 2025-12-04T09:42:48.7768298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7769040Z outputs = self.bert( 2025-12-04T09:42:48.7769756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7770512Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7771367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7772159Z layer_outputs = layer_module( 2025-12-04T09:42:48.7772972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7773696Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7774437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7775154Z return func(*args, **kwargs) 2025-12-04T09:42:48.7776045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7776887Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7777638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7778484Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7779368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7780279Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7781128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7781992Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7782696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7783336Z return self.act(input) 2025-12-04T09:42:48.7783544Z 2025-12-04T09:42:48.7783736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7784416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7785054Z res = mod(**inputs) 2025-12-04T09:42:48.7785769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7786523Z outputs = self.bert( 2025-12-04T09:42:48.7787237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7787995Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7788762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7789522Z layer_outputs = layer_module( 2025-12-04T09:42:48.7790148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7790802Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7791505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7792180Z return func(*args, **kwargs) 2025-12-04T09:42:48.7792928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7793712Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7794454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7795178Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7795996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7796933Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7797812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7798606Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7798847Z 2025-12-04T09:42:48.7799020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7799662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7800255Z res = mod(**inputs) 2025-12-04T09:42:48.7800986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7801752Z outputs = self.bert( 2025-12-04T09:42:48.7802560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7803341Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7804105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7804953Z layer_outputs = layer_module( 2025-12-04T09:42:48.7805632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7806302Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7807011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7807725Z return func(*args, **kwargs) 2025-12-04T09:42:48.7808501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7809326Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7810110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7810858Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7811948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7813043Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7813971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.7814815Z return input_tensor + hidden_states 2025-12-04T09:42:48.7815057Z 2025-12-04T09:42:48.7815251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7815953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7816561Z res = mod(**inputs) 2025-12-04T09:42:48.7817316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7818124Z outputs = self.bert( 2025-12-04T09:42:48.7818865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7819685Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7820496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7821314Z layer_outputs = layer_module( 2025-12-04T09:42:48.7822024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7822733Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7823466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7824174Z return func(*args, **kwargs) 2025-12-04T09:42:48.7824962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7825790Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7826529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7827242Z return func(*args, **kwargs) 2025-12-04T09:42:48.7828028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7828821Z self_outputs = self.self( 2025-12-04T09:42:48.7829498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7830255Z return func(*args, **kwargs) 2025-12-04T09:42:48.7831037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7831925Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7832178Z 2025-12-04T09:42:48.7832365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7833061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7833669Z res = mod(**inputs) 2025-12-04T09:42:48.7834438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7835234Z outputs = self.bert( 2025-12-04T09:42:48.7835998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7836835Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7837635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7838440Z layer_outputs = layer_module( 2025-12-04T09:42:48.7839159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7839865Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7840553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7841242Z return func(*args, **kwargs) 2025-12-04T09:42:48.7842009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7842807Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7843541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7844220Z return func(*args, **kwargs) 2025-12-04T09:42:48.7844979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7845746Z self_outputs = self.self( 2025-12-04T09:42:48.7846418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7847112Z return func(*args, **kwargs) 2025-12-04T09:42:48.7847864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7848668Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7848921Z 2025-12-04T09:42:48.7849104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7849797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7850398Z res = mod(**inputs) 2025-12-04T09:42:48.7851195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7852203Z outputs = self.bert( 2025-12-04T09:42:48.7853012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7853793Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7854591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7855374Z layer_outputs = layer_module( 2025-12-04T09:42:48.7855993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7856660Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7857443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7858140Z return func(*args, **kwargs) 2025-12-04T09:42:48.7858952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7859762Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7860552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7861222Z return func(*args, **kwargs) 2025-12-04T09:42:48.7861960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7862723Z self_outputs = self.self( 2025-12-04T09:42:48.7863365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7864022Z return func(*args, **kwargs) 2025-12-04T09:42:48.7864755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7865581Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7865818Z 2025-12-04T09:42:48.7865955Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7866295Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7866688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7867325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7867881Z res = mod(**inputs) 2025-12-04T09:42:48.7868608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7869365Z outputs = self.bert( 2025-12-04T09:42:48.7870084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7870851Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7871672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7872706Z layer_outputs = layer_module( 2025-12-04T09:42:48.7873383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7873512Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7874019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7874130Z return func(*args, **kwargs) 2025-12-04T09:42:48.7874685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7874816Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7875286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7875401Z return func(*args, **kwargs) 2025-12-04T09:42:48.7876002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7876240Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7876822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7876958Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7876966Z 2025-12-04T09:42:48.7877138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7877623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7877728Z res = mod(**inputs) 2025-12-04T09:42:48.7878338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7878518Z outputs = self.bert( 2025-12-04T09:42:48.7879110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7879228Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7879785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7879898Z layer_outputs = layer_module( 2025-12-04T09:42:48.7880326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7880448Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7880858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7880964Z return func(*args, **kwargs) 2025-12-04T09:42:48.7881510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7881696Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7882189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7882309Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7882883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7883047Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7883600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7883739Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7883747Z 2025-12-04T09:42:48.7883925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7884290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7884391Z res = mod(**inputs) 2025-12-04T09:42:48.7884951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7885064Z outputs = self.bert( 2025-12-04T09:42:48.7885611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7885736Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7886281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7886392Z layer_outputs = layer_module( 2025-12-04T09:42:48.7886818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7886954Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7887431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7887551Z return func(*args, **kwargs) 2025-12-04T09:42:48.7888109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7888253Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7888756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7888917Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7889557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7889775Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7890392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7890586Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7891001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7891128Z return self.act(input) 2025-12-04T09:42:48.7891136Z 2025-12-04T09:42:48.7891458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7891847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7891970Z res = mod(**inputs) 2025-12-04T09:42:48.7892579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7892695Z outputs = self.bert( 2025-12-04T09:42:48.7893283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7893403Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7893978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7894090Z layer_outputs = layer_module( 2025-12-04T09:42:48.7894508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7894634Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7895098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7895210Z return func(*args, **kwargs) 2025-12-04T09:42:48.7895761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7895901Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7896398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7896515Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7897112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7897340Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7897904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7898045Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7898053Z 2025-12-04T09:42:48.7898236Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7898622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7898727Z res = mod(**inputs) 2025-12-04T09:42:48.7899296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7899408Z outputs = self.bert( 2025-12-04T09:42:48.7899979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7900108Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7900895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7901044Z layer_outputs = layer_module( 2025-12-04T09:42:48.7901599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7901797Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7902356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7902466Z return func(*args, **kwargs) 2025-12-04T09:42:48.7902944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7903074Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7903523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7903629Z return func(*args, **kwargs) 2025-12-04T09:42:48.7904242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7904358Z self_outputs = self.self( 2025-12-04T09:42:48.7904794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7904943Z return func(*args, **kwargs) 2025-12-04T09:42:48.7905506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7905642Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7905650Z 2025-12-04T09:42:48.7905825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7906206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7906317Z res = mod(**inputs) 2025-12-04T09:42:48.7906903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7907014Z outputs = self.bert( 2025-12-04T09:42:48.7907567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7907696Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7908308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7908430Z layer_outputs = layer_module( 2025-12-04T09:42:48.7908884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7909015Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7909510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7909637Z return func(*args, **kwargs) 2025-12-04T09:42:48.7910200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7910331Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7910824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7910940Z return func(*args, **kwargs) 2025-12-04T09:42:48.7911526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7911653Z self_outputs = self.self( 2025-12-04T09:42:48.7912145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7912267Z return func(*args, **kwargs) 2025-12-04T09:42:48.7912906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7913038Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7913045Z 2025-12-04T09:42:48.7913244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7913701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7913817Z res = mod(**inputs) 2025-12-04T09:42:48.7914435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7914546Z outputs = self.bert( 2025-12-04T09:42:48.7915163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7915292Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7915914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7916046Z layer_outputs = layer_module( 2025-12-04T09:42:48.7916495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7916670Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7917163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7917277Z return func(*args, **kwargs) 2025-12-04T09:42:48.7917890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7918031Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7918529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7918637Z return func(*args, **kwargs) 2025-12-04T09:42:48.7919235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7919365Z self_outputs = self.self( 2025-12-04T09:42:48.7919869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7919980Z return func(*args, **kwargs) 2025-12-04T09:42:48.7920591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7920735Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7920743Z 2025-12-04T09:42:48.7920895Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7921031Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7921222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7921636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7921746Z res = mod(**inputs) 2025-12-04T09:42:48.7922367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7922496Z outputs = self.bert( 2025-12-04T09:42:48.7923094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7923232Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7923826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7923950Z layer_outputs = layer_module( 2025-12-04T09:42:48.7924431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7924567Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7925126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7925244Z return func(*args, **kwargs) 2025-12-04T09:42:48.7925884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7926041Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7926575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7926690Z return func(*args, **kwargs) 2025-12-04T09:42:48.7927311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7927549Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7928168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7928315Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7928323Z 2025-12-04T09:42:48.7928518Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7929519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7929626Z res = mod(**inputs) 2025-12-04T09:42:48.7930242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7930354Z outputs = self.bert( 2025-12-04T09:42:48.7930968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7931102Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7931842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7931969Z layer_outputs = layer_module( 2025-12-04T09:42:48.7932448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7932586Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7933100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7933214Z return func(*args, **kwargs) 2025-12-04T09:42:48.7933823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7933982Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7934532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7934670Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7935366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7935561Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7936189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7936335Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7936343Z 2025-12-04T09:42:48.7936528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7936951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7937058Z res = mod(**inputs) 2025-12-04T09:42:48.7937693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7937855Z outputs = self.bert( 2025-12-04T09:42:48.7938485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7938628Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7939294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7939461Z layer_outputs = layer_module( 2025-12-04T09:42:48.7939907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7940025Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7940500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7940606Z return func(*args, **kwargs) 2025-12-04T09:42:48.7941146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7941307Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7941808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7941970Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7942593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7942761Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7943325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.7943509Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.7943928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.7944033Z return self.act(input) 2025-12-04T09:42:48.7944041Z 2025-12-04T09:42:48.7944218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7944569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7944668Z res = mod(**inputs) 2025-12-04T09:42:48.7945227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7945338Z outputs = self.bert( 2025-12-04T09:42:48.7945887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7946007Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7946554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7946672Z layer_outputs = layer_module( 2025-12-04T09:42:48.7947089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7947222Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7947696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7947802Z return func(*args, **kwargs) 2025-12-04T09:42:48.7948350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7948493Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7948976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7949091Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7949766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7949999Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7950593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.7950751Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7950758Z 2025-12-04T09:42:48.7950938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7951318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7951416Z res = mod(**inputs) 2025-12-04T09:42:48.7951981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7952086Z outputs = self.bert( 2025-12-04T09:42:48.7952639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7952771Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7953340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7953483Z layer_outputs = layer_module( 2025-12-04T09:42:48.7953909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7954034Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7954499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7954608Z return func(*args, **kwargs) 2025-12-04T09:42:48.7955169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7955310Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7955813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7955945Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7956545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.7956773Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.7957342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.7957465Z return input_tensor + hidden_states 2025-12-04T09:42:48.7957472Z 2025-12-04T09:42:48.7957659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7958029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7958124Z res = mod(**inputs) 2025-12-04T09:42:48.7958686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7958794Z outputs = self.bert( 2025-12-04T09:42:48.7959349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7959470Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7960032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7960155Z layer_outputs = layer_module( 2025-12-04T09:42:48.7960591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7960719Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7961259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7961370Z return func(*args, **kwargs) 2025-12-04T09:42:48.7962010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7962183Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7962636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7962756Z return func(*args, **kwargs) 2025-12-04T09:42:48.7963298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7963409Z self_outputs = self.self( 2025-12-04T09:42:48.7963956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7964060Z return func(*args, **kwargs) 2025-12-04T09:42:48.7964640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.7964807Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.7964815Z 2025-12-04T09:42:48.7965002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7965393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7965495Z res = mod(**inputs) 2025-12-04T09:42:48.7966072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7966186Z outputs = self.bert( 2025-12-04T09:42:48.7966756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7966886Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7967450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7967573Z layer_outputs = layer_module( 2025-12-04T09:42:48.7968017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7968142Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7968612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7968719Z return func(*args, **kwargs) 2025-12-04T09:42:48.7969283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7969427Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7969893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7970006Z return func(*args, **kwargs) 2025-12-04T09:42:48.7970594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7970714Z self_outputs = self.self( 2025-12-04T09:42:48.7971203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7971416Z return func(*args, **kwargs) 2025-12-04T09:42:48.7972001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.7972150Z key_layer = self.key(current_states) 2025-12-04T09:42:48.7972159Z 2025-12-04T09:42:48.7972507Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7973041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7973148Z res = mod(**inputs) 2025-12-04T09:42:48.7973753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7973933Z outputs = self.bert( 2025-12-04T09:42:48.7974613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7974763Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7975387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7975501Z layer_outputs = layer_module( 2025-12-04T09:42:48.7975942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7976069Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7976516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7976635Z return func(*args, **kwargs) 2025-12-04T09:42:48.7977251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7977390Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7977866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7977978Z return func(*args, **kwargs) 2025-12-04T09:42:48.7978545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.7978655Z self_outputs = self.self( 2025-12-04T09:42:48.7979126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7979238Z return func(*args, **kwargs) 2025-12-04T09:42:48.7979781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.7979919Z value_layer = self.value(current_states) 2025-12-04T09:42:48.7979927Z 2025-12-04T09:42:48.7980055Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7980186Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.7980364Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7980742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7980841Z res = mod(**inputs) 2025-12-04T09:42:48.7981419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7981527Z outputs = self.bert( 2025-12-04T09:42:48.7982086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7982204Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7982763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7982887Z layer_outputs = layer_module( 2025-12-04T09:42:48.7983315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7983453Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7983917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7984024Z return func(*args, **kwargs) 2025-12-04T09:42:48.7984620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.7984752Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.7985200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7985347Z return func(*args, **kwargs) 2025-12-04T09:42:48.7985913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.7986139Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.7986691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.7986822Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7986829Z 2025-12-04T09:42:48.7987005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7987365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7987468Z res = mod(**inputs) 2025-12-04T09:42:48.7988037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7988173Z outputs = self.bert( 2025-12-04T09:42:48.7988745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7988861Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7989407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7989524Z layer_outputs = layer_module( 2025-12-04T09:42:48.7989935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7990067Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7990520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7990626Z return func(*args, **kwargs) 2025-12-04T09:42:48.7991193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7991334Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7991829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7991959Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7992579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7992763Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.7993319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.7993453Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.7993465Z 2025-12-04T09:42:48.7993654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.7994036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.7994144Z res = mod(**inputs) 2025-12-04T09:42:48.7994693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.7994795Z outputs = self.bert( 2025-12-04T09:42:48.7995351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.7995465Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.7996065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.7996178Z layer_outputs = layer_module( 2025-12-04T09:42:48.7996623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.7996760Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.7997232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.7997353Z return func(*args, **kwargs) 2025-12-04T09:42:48.7997898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.7998031Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.7998538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.7998654Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.7999242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.7999467Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8000002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8000189Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8000582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8000688Z return self.act(input) 2025-12-04T09:42:48.8000695Z 2025-12-04T09:42:48.8000876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8001229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8001324Z res = mod(**inputs) 2025-12-04T09:42:48.8001892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8001996Z outputs = self.bert( 2025-12-04T09:42:48.8002542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8002652Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8003189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8003324Z layer_outputs = layer_module( 2025-12-04T09:42:48.8003724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8003857Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8004303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8004401Z return func(*args, **kwargs) 2025-12-04T09:42:48.8004948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8005078Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8005557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8005683Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8006268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8006498Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8007066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8007195Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8007202Z 2025-12-04T09:42:48.8007418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8007787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8007913Z res = mod(**inputs) 2025-12-04T09:42:48.8008462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8008566Z outputs = self.bert( 2025-12-04T09:42:48.8009123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8009239Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8009804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8009924Z layer_outputs = layer_module( 2025-12-04T09:42:48.8010350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8010511Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8010980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8011104Z return func(*args, **kwargs) 2025-12-04T09:42:48.8011854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8012012Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8012537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8012647Z return func(*args, **kwargs) 2025-12-04T09:42:48.8013225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8013343Z self_outputs = self.self( 2025-12-04T09:42:48.8013806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8013912Z return func(*args, **kwargs) 2025-12-04T09:42:48.8014484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8014612Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8014620Z 2025-12-04T09:42:48.8014805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8015189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8015292Z res = mod(**inputs) 2025-12-04T09:42:48.8015872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8015975Z outputs = self.bert( 2025-12-04T09:42:48.8016518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8016640Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8017207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8017326Z layer_outputs = layer_module( 2025-12-04T09:42:48.8017752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8017876Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8018402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8018515Z return func(*args, **kwargs) 2025-12-04T09:42:48.8019088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8019256Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8019749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8019873Z return func(*args, **kwargs) 2025-12-04T09:42:48.8020441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8020558Z self_outputs = self.self( 2025-12-04T09:42:48.8021038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8021163Z return func(*args, **kwargs) 2025-12-04T09:42:48.8021739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8021864Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8021874Z 2025-12-04T09:42:48.8022110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8022493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8022605Z res = mod(**inputs) 2025-12-04T09:42:48.8023193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8023298Z outputs = self.bert( 2025-12-04T09:42:48.8023872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8024009Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8024533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8024644Z layer_outputs = layer_module( 2025-12-04T09:42:48.8025064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8025192Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8025659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8025765Z return func(*args, **kwargs) 2025-12-04T09:42:48.8026294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8026432Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8026883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8027005Z return func(*args, **kwargs) 2025-12-04T09:42:48.8027566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8027678Z self_outputs = self.self( 2025-12-04T09:42:48.8028141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8028246Z return func(*args, **kwargs) 2025-12-04T09:42:48.8028803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8028936Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8028942Z 2025-12-04T09:42:48.8029067Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8029205Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8029382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8029788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8029900Z res = mod(**inputs) 2025-12-04T09:42:48.8030455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8030588Z outputs = self.bert( 2025-12-04T09:42:48.8031179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8031297Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8031861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8031975Z layer_outputs = layer_module( 2025-12-04T09:42:48.8032390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8032526Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8032985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8033101Z return func(*args, **kwargs) 2025-12-04T09:42:48.8033688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8033822Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8034281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8034386Z return func(*args, **kwargs) 2025-12-04T09:42:48.8034936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8035162Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8035721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8035862Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8035873Z 2025-12-04T09:42:48.8036051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8036431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8036536Z res = mod(**inputs) 2025-12-04T09:42:48.8037081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8037186Z outputs = self.bert( 2025-12-04T09:42:48.8037712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8037825Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8038336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8038446Z layer_outputs = layer_module( 2025-12-04T09:42:48.8038847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8038974Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8039399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8039503Z return func(*args, **kwargs) 2025-12-04T09:42:48.8040009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8040128Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8040605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8040752Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8041335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8041532Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8042097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8042228Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8042236Z 2025-12-04T09:42:48.8042401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8042746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8042847Z res = mod(**inputs) 2025-12-04T09:42:48.8043374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8043481Z outputs = self.bert( 2025-12-04T09:42:48.8044014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8044152Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8044690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8044796Z layer_outputs = layer_module( 2025-12-04T09:42:48.8045192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8045311Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8045743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8045851Z return func(*args, **kwargs) 2025-12-04T09:42:48.8046368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8046495Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8046994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8047111Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8047723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8047889Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8048426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8048616Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8049013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8049128Z return self.act(input) 2025-12-04T09:42:48.8049134Z 2025-12-04T09:42:48.8049309Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8049675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8049781Z res = mod(**inputs) 2025-12-04T09:42:48.8050324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8050428Z outputs = self.bert( 2025-12-04T09:42:48.8050976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8051093Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8051833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8051966Z layer_outputs = layer_module( 2025-12-04T09:42:48.8052411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8052639Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8053161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8053286Z return func(*args, **kwargs) 2025-12-04T09:42:48.8053892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8054039Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8054574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8054706Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8055381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8055645Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8056292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8056442Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8056450Z 2025-12-04T09:42:48.8056641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8057046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8057164Z res = mod(**inputs) 2025-12-04T09:42:48.8057783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8057907Z outputs = self.bert( 2025-12-04T09:42:48.8058502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8058631Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8059244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8059368Z layer_outputs = layer_module( 2025-12-04T09:42:48.8059835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8059974Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8060437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8060550Z return func(*args, **kwargs) 2025-12-04T09:42:48.8061089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8061212Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8061703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8061818Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8062401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8062606Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8063114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.8063235Z return input_tensor + hidden_states 2025-12-04T09:42:48.8063242Z 2025-12-04T09:42:48.8063407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8063789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8063879Z res = mod(**inputs) 2025-12-04T09:42:48.8064410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8064550Z outputs = self.bert( 2025-12-04T09:42:48.8065097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8065209Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8065734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8065836Z layer_outputs = layer_module( 2025-12-04T09:42:48.8066259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8066380Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8066824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8066967Z return func(*args, **kwargs) 2025-12-04T09:42:48.8067510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8067641Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8068103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8068208Z return func(*args, **kwargs) 2025-12-04T09:42:48.8068742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8068852Z self_outputs = self.self( 2025-12-04T09:42:48.8069309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8069419Z return func(*args, **kwargs) 2025-12-04T09:42:48.8069938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8070074Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8070080Z 2025-12-04T09:42:48.8070262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8070603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8070704Z res = mod(**inputs) 2025-12-04T09:42:48.8071233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8071327Z outputs = self.bert( 2025-12-04T09:42:48.8071861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8071971Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8072662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8072787Z layer_outputs = layer_module( 2025-12-04T09:42:48.8073181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8073311Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8073754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8073861Z return func(*args, **kwargs) 2025-12-04T09:42:48.8074895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8075375Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8075692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8075771Z return func(*args, **kwargs) 2025-12-04T09:42:48.8076120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8076235Z self_outputs = self.self( 2025-12-04T09:42:48.8076482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8076554Z return func(*args, **kwargs) 2025-12-04T09:42:48.8076970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8077055Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8077062Z 2025-12-04T09:42:48.8077186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8077390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8077466Z res = mod(**inputs) 2025-12-04T09:42:48.8077752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8077859Z outputs = self.bert( 2025-12-04T09:42:48.8078144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8078216Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8078492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8078570Z layer_outputs = layer_module( 2025-12-04T09:42:48.8078788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8078874Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8079109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8079179Z return func(*args, **kwargs) 2025-12-04T09:42:48.8079461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8079541Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8079775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8079848Z return func(*args, **kwargs) 2025-12-04T09:42:48.8080125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8080201Z self_outputs = self.self( 2025-12-04T09:42:48.8080431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8080498Z return func(*args, **kwargs) 2025-12-04T09:42:48.8080782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8080862Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8080867Z 2025-12-04T09:42:48.8080952Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8081026Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8081127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8081332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8081395Z res = mod(**inputs) 2025-12-04T09:42:48.8081694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8081767Z outputs = self.bert( 2025-12-04T09:42:48.8082043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8082146Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8082441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8082511Z layer_outputs = layer_module( 2025-12-04T09:42:48.8082733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8082809Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8083040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8083113Z return func(*args, **kwargs) 2025-12-04T09:42:48.8083390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8083488Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8083723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8083806Z return func(*args, **kwargs) 2025-12-04T09:42:48.8084095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8084221Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8084508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8084589Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8084593Z 2025-12-04T09:42:48.8084698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8084902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8084966Z res = mod(**inputs) 2025-12-04T09:42:48.8085262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8085331Z outputs = self.bert( 2025-12-04T09:42:48.8085616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8085696Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8085981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8086051Z layer_outputs = layer_module( 2025-12-04T09:42:48.8086279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8086354Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8086600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8086669Z return func(*args, **kwargs) 2025-12-04T09:42:48.8086951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8087040Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8087303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8087380Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8087699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8087823Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8088111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8088215Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8088219Z 2025-12-04T09:42:48.8088324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8088544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8088609Z res = mod(**inputs) 2025-12-04T09:42:48.8088899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8088963Z outputs = self.bert( 2025-12-04T09:42:48.8089244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8089323Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8089606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8089684Z layer_outputs = layer_module( 2025-12-04T09:42:48.8089974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8090054Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8090307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8090374Z return func(*args, **kwargs) 2025-12-04T09:42:48.8090665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8090757Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8091020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8091105Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8091577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8091708Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8092073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8092198Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8092457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8092532Z return self.act(input) 2025-12-04T09:42:48.8092537Z 2025-12-04T09:42:48.8092648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8092871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8092939Z res = mod(**inputs) 2025-12-04T09:42:48.8093247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8093330Z outputs = self.bert( 2025-12-04T09:42:48.8093688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8093768Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8094043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8094111Z layer_outputs = layer_module( 2025-12-04T09:42:48.8094334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8094472Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8094713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8094787Z return func(*args, **kwargs) 2025-12-04T09:42:48.8095085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8095191Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8095444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8095518Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8095835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8095967Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8096258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8096338Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8096359Z 2025-12-04T09:42:48.8096462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8096665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8096728Z res = mod(**inputs) 2025-12-04T09:42:48.8097019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8097084Z outputs = self.bert( 2025-12-04T09:42:48.8097363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8097440Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8097722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8097792Z layer_outputs = layer_module( 2025-12-04T09:42:48.8098018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8098095Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8098339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8098405Z return func(*args, **kwargs) 2025-12-04T09:42:48.8098686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8098773Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8099008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8099073Z return func(*args, **kwargs) 2025-12-04T09:42:48.8099358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8099428Z self_outputs = self.self( 2025-12-04T09:42:48.8099674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8099740Z return func(*args, **kwargs) 2025-12-04T09:42:48.8100028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8100114Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8100118Z 2025-12-04T09:42:48.8100219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8100415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8100521Z res = mod(**inputs) 2025-12-04T09:42:48.8100807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8100897Z outputs = self.bert( 2025-12-04T09:42:48.8101179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8101265Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8101553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8101620Z layer_outputs = layer_module( 2025-12-04T09:42:48.8101850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8101921Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8102143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8102213Z return func(*args, **kwargs) 2025-12-04T09:42:48.8102474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8102572Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8102795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8102858Z return func(*args, **kwargs) 2025-12-04T09:42:48.8103129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8103193Z self_outputs = self.self( 2025-12-04T09:42:48.8103413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8103482Z return func(*args, **kwargs) 2025-12-04T09:42:48.8103745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8103825Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8103830Z 2025-12-04T09:42:48.8103925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8104106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8104174Z res = mod(**inputs) 2025-12-04T09:42:48.8104440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8104505Z outputs = self.bert( 2025-12-04T09:42:48.8104765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8104830Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8105101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8105165Z layer_outputs = layer_module( 2025-12-04T09:42:48.8105371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8105450Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8105668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8105734Z return func(*args, **kwargs) 2025-12-04T09:42:48.8105997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8106070Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8106317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8106381Z return func(*args, **kwargs) 2025-12-04T09:42:48.8106650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8106731Z self_outputs = self.self( 2025-12-04T09:42:48.8106967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8107035Z return func(*args, **kwargs) 2025-12-04T09:42:48.8107301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8107375Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8107378Z 2025-12-04T09:42:48.8107459Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8107532Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8107634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8107816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8107874Z res = mod(**inputs) 2025-12-04T09:42:48.8108150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8108228Z outputs = self.bert( 2025-12-04T09:42:48.8108494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8108567Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8108834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8108905Z layer_outputs = layer_module( 2025-12-04T09:42:48.8109112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8109184Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8109412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8109477Z return func(*args, **kwargs) 2025-12-04T09:42:48.8109749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8109824Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8110045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8110112Z return func(*args, **kwargs) 2025-12-04T09:42:48.8110378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8110498Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8110773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8110852Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8110856Z 2025-12-04T09:42:48.8110958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8111142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8111202Z res = mod(**inputs) 2025-12-04T09:42:48.8111477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8111539Z outputs = self.bert( 2025-12-04T09:42:48.8111811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8111878Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8112151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8112228Z layer_outputs = layer_module( 2025-12-04T09:42:48.8112461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8112549Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8112812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8112878Z return func(*args, **kwargs) 2025-12-04T09:42:48.8113151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8113228Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8113471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8113551Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8113859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8113995Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8114261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8114337Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8114340Z 2025-12-04T09:42:48.8114442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8114624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8114683Z res = mod(**inputs) 2025-12-04T09:42:48.8114959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8115020Z outputs = self.bert( 2025-12-04T09:42:48.8115294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8115364Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8115633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8115704Z layer_outputs = layer_module( 2025-12-04T09:42:48.8115909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8115988Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8116212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8116278Z return func(*args, **kwargs) 2025-12-04T09:42:48.8116554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8116633Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8116875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8116953Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8117248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8117348Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8117614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8117719Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8117941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8118010Z return self.act(input) 2025-12-04T09:42:48.8118029Z 2025-12-04T09:42:48.8118134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8118319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8118394Z res = mod(**inputs) 2025-12-04T09:42:48.8118680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8118742Z outputs = self.bert( 2025-12-04T09:42:48.8119016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8119089Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8119363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8119434Z layer_outputs = layer_module( 2025-12-04T09:42:48.8119642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8119738Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8119977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8120042Z return func(*args, **kwargs) 2025-12-04T09:42:48.8120323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8120404Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8120650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8120728Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8121033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8121160Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8121433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8121509Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8121512Z 2025-12-04T09:42:48.8121616Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8121798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8121857Z res = mod(**inputs) 2025-12-04T09:42:48.8122141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8122203Z outputs = self.bert( 2025-12-04T09:42:48.8122478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8122548Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8122822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8122896Z layer_outputs = layer_module( 2025-12-04T09:42:48.8123105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8123179Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8123416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8123481Z return func(*args, **kwargs) 2025-12-04T09:42:48.8123777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8123857Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8124119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8124201Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8124515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8124651Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8124945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.8125021Z return input_tensor + hidden_states 2025-12-04T09:42:48.8125025Z 2025-12-04T09:42:48.8125139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8125330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8125392Z res = mod(**inputs) 2025-12-04T09:42:48.8125713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8125783Z outputs = self.bert( 2025-12-04T09:42:48.8126068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8126139Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8126416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8126495Z layer_outputs = layer_module( 2025-12-04T09:42:48.8126708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8126792Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8127023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8127094Z return func(*args, **kwargs) 2025-12-04T09:42:48.8127379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8127461Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8127695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8127773Z return func(*args, **kwargs) 2025-12-04T09:42:48.8128054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8128132Z self_outputs = self.self( 2025-12-04T09:42:48.8128369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8128436Z return func(*args, **kwargs) 2025-12-04T09:42:48.8128725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8128809Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8128813Z 2025-12-04T09:42:48.8128925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8129117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8129185Z res = mod(**inputs) 2025-12-04T09:42:48.8129476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8129542Z outputs = self.bert( 2025-12-04T09:42:48.8129842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8129922Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8130217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8130296Z layer_outputs = layer_module( 2025-12-04T09:42:48.8130526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8130603Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8130845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8130910Z return func(*args, **kwargs) 2025-12-04T09:42:48.8131193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8131373Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8131646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8131756Z return func(*args, **kwargs) 2025-12-04T09:42:48.8132075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8132154Z self_outputs = self.self( 2025-12-04T09:42:48.8132425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8132499Z return func(*args, **kwargs) 2025-12-04T09:42:48.8132843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8132926Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8132932Z 2025-12-04T09:42:48.8133041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8133259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8133328Z res = mod(**inputs) 2025-12-04T09:42:48.8133633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8133712Z outputs = self.bert( 2025-12-04T09:42:48.8134014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8134097Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8134394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8134459Z layer_outputs = layer_module( 2025-12-04T09:42:48.8134678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8134751Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8134982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8135048Z return func(*args, **kwargs) 2025-12-04T09:42:48.8135328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8135414Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8135645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8135710Z return func(*args, **kwargs) 2025-12-04T09:42:48.8135994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8136077Z self_outputs = self.self( 2025-12-04T09:42:48.8136317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8136402Z return func(*args, **kwargs) 2025-12-04T09:42:48.8136686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8136794Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8136798Z 2025-12-04T09:42:48.8136877Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8136960Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8137061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8137254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8137321Z res = mod(**inputs) 2025-12-04T09:42:48.8137602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8137664Z outputs = self.bert( 2025-12-04T09:42:48.8137956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8138044Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8138325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8138392Z layer_outputs = layer_module( 2025-12-04T09:42:48.8138609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8138689Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8138925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8138993Z return func(*args, **kwargs) 2025-12-04T09:42:48.8139279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8139359Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8139605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8139672Z return func(*args, **kwargs) 2025-12-04T09:42:48.8139952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8140087Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8140366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8140452Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8140457Z 2025-12-04T09:42:48.8140558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8140750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8140822Z res = mod(**inputs) 2025-12-04T09:42:48.8141106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8141169Z outputs = self.bert( 2025-12-04T09:42:48.8141456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8141528Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8141818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8141885Z layer_outputs = layer_module( 2025-12-04T09:42:48.8142124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8142211Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8142445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8142542Z return func(*args, **kwargs) 2025-12-04T09:42:48.8142838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8142922Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8143185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8143261Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8143573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8143683Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8143964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8144076Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8144080Z 2025-12-04T09:42:48.8144182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8144377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8144448Z res = mod(**inputs) 2025-12-04T09:42:48.8144732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8144802Z outputs = self.bert( 2025-12-04T09:42:48.8145083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8145161Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8145446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8145519Z layer_outputs = layer_module( 2025-12-04T09:42:48.8145735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8145818Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8146054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8146129Z return func(*args, **kwargs) 2025-12-04T09:42:48.8146411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8146489Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8146751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8146824Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8147143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8147246Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8147537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8147654Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8147867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8147940Z return self.act(input) 2025-12-04T09:42:48.8147944Z 2025-12-04T09:42:48.8148063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8148259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8148329Z res = mod(**inputs) 2025-12-04T09:42:48.8148634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8148709Z outputs = self.bert( 2025-12-04T09:42:48.8148994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8149065Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8149338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8149405Z layer_outputs = layer_module( 2025-12-04T09:42:48.8149610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8149691Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8149914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8149997Z return func(*args, **kwargs) 2025-12-04T09:42:48.8150278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8150353Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8150601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8150669Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8150970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8151101Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8151375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8151457Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8151460Z 2025-12-04T09:42:48.8151554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8151741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8151807Z res = mod(**inputs) 2025-12-04T09:42:48.8152083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8152151Z outputs = self.bert( 2025-12-04T09:42:48.8152423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8152493Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8152781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8152849Z layer_outputs = layer_module( 2025-12-04T09:42:48.8153066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8153149Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8153382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8153454Z return func(*args, **kwargs) 2025-12-04T09:42:48.8153737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8153811Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8154069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8154138Z return func(*args, **kwargs) 2025-12-04T09:42:48.8154412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8154503Z self_outputs = self.self( 2025-12-04T09:42:48.8154753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8154828Z return func(*args, **kwargs) 2025-12-04T09:42:48.8155105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8155182Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8155185Z 2025-12-04T09:42:48.8155289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8155476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8155544Z res = mod(**inputs) 2025-12-04T09:42:48.8155816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8155901Z outputs = self.bert( 2025-12-04T09:42:48.8156177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8156245Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8156516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8156590Z layer_outputs = layer_module( 2025-12-04T09:42:48.8156800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8156880Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8157112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8157177Z return func(*args, **kwargs) 2025-12-04T09:42:48.8157470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8157554Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8157802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8157870Z return func(*args, **kwargs) 2025-12-04T09:42:48.8158156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8158232Z self_outputs = self.self( 2025-12-04T09:42:48.8158473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8158540Z return func(*args, **kwargs) 2025-12-04T09:42:48.8158840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8158918Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8158922Z 2025-12-04T09:42:48.8159029Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8159218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8159290Z res = mod(**inputs) 2025-12-04T09:42:48.8159573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8159634Z outputs = self.bert( 2025-12-04T09:42:48.8159912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8159997Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8160275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8160382Z layer_outputs = layer_module( 2025-12-04T09:42:48.8160613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8160690Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8160935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8161002Z return func(*args, **kwargs) 2025-12-04T09:42:48.8161290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8161368Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8161607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8161680Z return func(*args, **kwargs) 2025-12-04T09:42:48.8161963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8162057Z self_outputs = self.self( 2025-12-04T09:42:48.8162305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8162372Z return func(*args, **kwargs) 2025-12-04T09:42:48.8162673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8162755Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8162758Z 2025-12-04T09:42:48.8162835Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8162920Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8163026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8163228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8163294Z res = mod(**inputs) 2025-12-04T09:42:48.8163585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8163658Z outputs = self.bert( 2025-12-04T09:42:48.8163948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8164017Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8164313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8164381Z layer_outputs = layer_module( 2025-12-04T09:42:48.8164611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8164687Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8164928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8165006Z return func(*args, **kwargs) 2025-12-04T09:42:48.8165295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8165379Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8165620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8165691Z return func(*args, **kwargs) 2025-12-04T09:42:48.8165986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8166126Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8166407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8166514Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8166517Z 2025-12-04T09:42:48.8166635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8166832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8166895Z res = mod(**inputs) 2025-12-04T09:42:48.8167178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8167254Z outputs = self.bert( 2025-12-04T09:42:48.8167534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8167612Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8167897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8167986Z layer_outputs = layer_module( 2025-12-04T09:42:48.8168211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8168285Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8168517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8168591Z return func(*args, **kwargs) 2025-12-04T09:42:48.8168869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8168957Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8169209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8169283Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8169595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8169702Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8169985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8170062Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8170065Z 2025-12-04T09:42:48.8170166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8170361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8170423Z res = mod(**inputs) 2025-12-04T09:42:48.8170709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8170780Z outputs = self.bert( 2025-12-04T09:42:48.8171072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8171156Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8171546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8171628Z layer_outputs = layer_module( 2025-12-04T09:42:48.8171874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8171959Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8172412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8172500Z return func(*args, **kwargs) 2025-12-04T09:42:48.8172817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8172959Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8173259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8173339Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8173676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8173776Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8174064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8174177Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8174386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8174464Z return self.act(input) 2025-12-04T09:42:48.8174492Z 2025-12-04T09:42:48.8174595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8174799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8174862Z res = mod(**inputs) 2025-12-04T09:42:48.8175144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8175216Z outputs = self.bert( 2025-12-04T09:42:48.8175499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8175568Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8175854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8175923Z layer_outputs = layer_module( 2025-12-04T09:42:48.8176147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8176224Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8176458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8176533Z return func(*args, **kwargs) 2025-12-04T09:42:48.8176810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8176896Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8177144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8177216Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8177528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8177660Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8177941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8178024Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8178027Z 2025-12-04T09:42:48.8178126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8178324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8178386Z res = mod(**inputs) 2025-12-04T09:42:48.8178691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8178765Z outputs = self.bert( 2025-12-04T09:42:48.8179052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8179152Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8179466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8179538Z layer_outputs = layer_module( 2025-12-04T09:42:48.8179767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8179844Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8180091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8180171Z return func(*args, **kwargs) 2025-12-04T09:42:48.8180462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8180554Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8180832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8180910Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8181232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8181361Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8181651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.8181726Z return input_tensor + hidden_states 2025-12-04T09:42:48.8181732Z 2025-12-04T09:42:48.8181834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8182035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8182110Z res = mod(**inputs) 2025-12-04T09:42:48.8182394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8182466Z outputs = self.bert( 2025-12-04T09:42:48.8182745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8182823Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8183107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8183176Z layer_outputs = layer_module( 2025-12-04T09:42:48.8183401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8183479Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8183725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8183796Z return func(*args, **kwargs) 2025-12-04T09:42:48.8184086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8184171Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8184408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8184476Z return func(*args, **kwargs) 2025-12-04T09:42:48.8184797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8184868Z self_outputs = self.self( 2025-12-04T09:42:48.8185114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8185203Z return func(*args, **kwargs) 2025-12-04T09:42:48.8185513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8185603Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8185607Z 2025-12-04T09:42:48.8185709Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8185912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8185977Z res = mod(**inputs) 2025-12-04T09:42:48.8186268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8186343Z outputs = self.bert( 2025-12-04T09:42:48.8186639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8186712Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8187027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8187097Z layer_outputs = layer_module( 2025-12-04T09:42:48.8187325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8187400Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8187639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8187712Z return func(*args, **kwargs) 2025-12-04T09:42:48.8188000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8188087Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8188328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8188397Z return func(*args, **kwargs) 2025-12-04T09:42:48.8188691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8188759Z self_outputs = self.self( 2025-12-04T09:42:48.8188999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8189072Z return func(*args, **kwargs) 2025-12-04T09:42:48.8189359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8189445Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8189449Z 2025-12-04T09:42:48.8189551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8189745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8189816Z res = mod(**inputs) 2025-12-04T09:42:48.8190104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8190168Z outputs = self.bert( 2025-12-04T09:42:48.8190466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8190539Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8190835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8190927Z layer_outputs = layer_module( 2025-12-04T09:42:48.8191156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8191242Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8191504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8191581Z return func(*args, **kwargs) 2025-12-04T09:42:48.8191890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8191976Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8192221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8192287Z return func(*args, **kwargs) 2025-12-04T09:42:48.8192578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8192656Z self_outputs = self.self( 2025-12-04T09:42:48.8192903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8193023Z return func(*args, **kwargs) 2025-12-04T09:42:48.8193321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8193401Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8193404Z 2025-12-04T09:42:48.8193494Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8193573Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8193682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8193881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8193947Z res = mod(**inputs) 2025-12-04T09:42:48.8194252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8194318Z outputs = self.bert( 2025-12-04T09:42:48.8194618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8194696Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8194976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8195051Z layer_outputs = layer_module( 2025-12-04T09:42:48.8195276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8195348Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8195585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8195649Z return func(*args, **kwargs) 2025-12-04T09:42:48.8195920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8196007Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8196235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8196307Z return func(*args, **kwargs) 2025-12-04T09:42:48.8196579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8196703Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8196986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8197082Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8197086Z 2025-12-04T09:42:48.8197194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8197382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8197462Z res = mod(**inputs) 2025-12-04T09:42:48.8197769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8197834Z outputs = self.bert( 2025-12-04T09:42:48.8198111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8198188Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8198465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8198544Z layer_outputs = layer_module( 2025-12-04T09:42:48.8198758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8198831Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8199093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8199158Z return func(*args, **kwargs) 2025-12-04T09:42:48.8199452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8199532Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8199775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8199855Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8200159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8200257Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8200541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8200623Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8200627Z 2025-12-04T09:42:48.8200733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8200921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8200982Z res = mod(**inputs) 2025-12-04T09:42:48.8201268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8201330Z outputs = self.bert( 2025-12-04T09:42:48.8201612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8201680Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8201954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8202033Z layer_outputs = layer_module( 2025-12-04T09:42:48.8202246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8202321Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8202565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8202630Z return func(*args, **kwargs) 2025-12-04T09:42:48.8202911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8203020Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8203266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8203365Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8203721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8203834Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8204111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8204220Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8204437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8204505Z return self.act(input) 2025-12-04T09:42:48.8204508Z 2025-12-04T09:42:48.8204615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8204805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8204870Z res = mod(**inputs) 2025-12-04T09:42:48.8205191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8205261Z outputs = self.bert( 2025-12-04T09:42:48.8205541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8205620Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8205899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8205981Z layer_outputs = layer_module( 2025-12-04T09:42:48.8206200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8206276Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8206530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8206601Z return func(*args, **kwargs) 2025-12-04T09:42:48.8206885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8206977Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8207233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8207324Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8207634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8207764Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8208053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8208137Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8208141Z 2025-12-04T09:42:48.8208250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8208446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8208511Z res = mod(**inputs) 2025-12-04T09:42:48.8208810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8208877Z outputs = self.bert( 2025-12-04T09:42:48.8209193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8209268Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8209557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8209656Z layer_outputs = layer_module( 2025-12-04T09:42:48.8209906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8209988Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8210236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8210302Z return func(*args, **kwargs) 2025-12-04T09:42:48.8210592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8210672Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8210912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8210988Z return func(*args, **kwargs) 2025-12-04T09:42:48.8211349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8211449Z self_outputs = self.self( 2025-12-04T09:42:48.8211710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8211780Z return func(*args, **kwargs) 2025-12-04T09:42:48.8212088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8212172Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8212176Z 2025-12-04T09:42:48.8212285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8212504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8212573Z res = mod(**inputs) 2025-12-04T09:42:48.8212889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8212961Z outputs = self.bert( 2025-12-04T09:42:48.8213274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8213356Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8213641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8213710Z layer_outputs = layer_module( 2025-12-04T09:42:48.8213936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8214012Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8214257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8214323Z return func(*args, **kwargs) 2025-12-04T09:42:48.8214609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8214698Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8214937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8215010Z return func(*args, **kwargs) 2025-12-04T09:42:48.8215297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8215363Z self_outputs = self.self( 2025-12-04T09:42:48.8215622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8215692Z return func(*args, **kwargs) 2025-12-04T09:42:48.8215977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8216084Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8216088Z 2025-12-04T09:42:48.8216210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8216417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8216485Z res = mod(**inputs) 2025-12-04T09:42:48.8216781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8216852Z outputs = self.bert( 2025-12-04T09:42:48.8217141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8217218Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8217506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8217595Z layer_outputs = layer_module( 2025-12-04T09:42:48.8217830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8217905Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8218148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8218223Z return func(*args, **kwargs) 2025-12-04T09:42:48.8218515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8218603Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8218846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8218913Z return func(*args, **kwargs) 2025-12-04T09:42:48.8219209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8219278Z self_outputs = self.self( 2025-12-04T09:42:48.8219522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8219599Z return func(*args, **kwargs) 2025-12-04T09:42:48.8219888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8219978Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8219982Z 2025-12-04T09:42:48.8220065Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8220146Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8220256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8220454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8220532Z res = mod(**inputs) 2025-12-04T09:42:48.8220824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8220890Z outputs = self.bert( 2025-12-04T09:42:48.8221188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8221259Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8221549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8221646Z layer_outputs = layer_module( 2025-12-04T09:42:48.8221872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8221959Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8222220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8222287Z return func(*args, **kwargs) 2025-12-04T09:42:48.8222596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8222678Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8222920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8222994Z return func(*args, **kwargs) 2025-12-04T09:42:48.8223285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8223417Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8223706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8223808Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8223812Z 2025-12-04T09:42:48.8223923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8224120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8224189Z res = mod(**inputs) 2025-12-04T09:42:48.8224476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8224540Z outputs = self.bert( 2025-12-04T09:42:48.8224834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8224905Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8225199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8225270Z layer_outputs = layer_module( 2025-12-04T09:42:48.8225490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8225574Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8225817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8225886Z return func(*args, **kwargs) 2025-12-04T09:42:48.8226184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8226266Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8226525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8226599Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8226908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8227017Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8227298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8227384Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8227388Z 2025-12-04T09:42:48.8227488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8227680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8227767Z res = mod(**inputs) 2025-12-04T09:42:48.8228052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8228137Z outputs = self.bert( 2025-12-04T09:42:48.8228434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8228528Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8228826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8228895Z layer_outputs = layer_module( 2025-12-04T09:42:48.8229119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8229204Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8229447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8229524Z return func(*args, **kwargs) 2025-12-04T09:42:48.8229822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8229924Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8230183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8230255Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8230559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8230665Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8230945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8231059Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8231266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8231336Z return self.act(input) 2025-12-04T09:42:48.8231339Z 2025-12-04T09:42:48.8231443Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8231636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8231705Z res = mod(**inputs) 2025-12-04T09:42:48.8231988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8232050Z outputs = self.bert( 2025-12-04T09:42:48.8232336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8232409Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8232686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8232765Z layer_outputs = layer_module( 2025-12-04T09:42:48.8232979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8233062Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8233296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8233362Z return func(*args, **kwargs) 2025-12-04T09:42:48.8233648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8233778Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8234080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8234158Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8234471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8234649Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8234931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8235010Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8235023Z 2025-12-04T09:42:48.8235124Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8235312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8235383Z res = mod(**inputs) 2025-12-04T09:42:48.8235671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8235738Z outputs = self.bert( 2025-12-04T09:42:48.8236034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8236129Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8236424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8236497Z layer_outputs = layer_module( 2025-12-04T09:42:48.8236721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8236807Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8237053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8237135Z return func(*args, **kwargs) 2025-12-04T09:42:48.8237432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8237525Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8237791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8237868Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8238184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8238322Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8238610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.8238696Z return input_tensor + hidden_states 2025-12-04T09:42:48.8238700Z 2025-12-04T09:42:48.8238805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8239013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8239085Z res = mod(**inputs) 2025-12-04T09:42:48.8239373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8239437Z outputs = self.bert( 2025-12-04T09:42:48.8239734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8239804Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8240092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8240182Z layer_outputs = layer_module( 2025-12-04T09:42:48.8240401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8240483Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8240741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8240839Z return func(*args, **kwargs) 2025-12-04T09:42:48.8241121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8241200Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8241441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8241506Z return func(*args, **kwargs) 2025-12-04T09:42:48.8241787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8241863Z self_outputs = self.self( 2025-12-04T09:42:48.8242096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8242190Z return func(*args, **kwargs) 2025-12-04T09:42:48.8242478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8242558Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8242562Z 2025-12-04T09:42:48.8242670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8242865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8242934Z res = mod(**inputs) 2025-12-04T09:42:48.8243229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8243291Z outputs = self.bert( 2025-12-04T09:42:48.8243575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8243646Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8243922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8244000Z layer_outputs = layer_module( 2025-12-04T09:42:48.8244212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8244290Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8244527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8244590Z return func(*args, **kwargs) 2025-12-04T09:42:48.8244873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8244949Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8245182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8245253Z return func(*args, **kwargs) 2025-12-04T09:42:48.8245530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8245601Z self_outputs = self.self( 2025-12-04T09:42:48.8245834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8245896Z return func(*args, **kwargs) 2025-12-04T09:42:48.8246202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8246280Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8246284Z 2025-12-04T09:42:48.8246389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8246599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8246660Z res = mod(**inputs) 2025-12-04T09:42:48.8246972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8247037Z outputs = self.bert( 2025-12-04T09:42:48.8247316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8247392Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8247676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8247750Z layer_outputs = layer_module( 2025-12-04T09:42:48.8247965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8248070Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8248313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8248383Z return func(*args, **kwargs) 2025-12-04T09:42:48.8248670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8248751Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8248986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8249060Z return func(*args, **kwargs) 2025-12-04T09:42:48.8249345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8249414Z self_outputs = self.self( 2025-12-04T09:42:48.8249659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8249728Z return func(*args, **kwargs) 2025-12-04T09:42:48.8250017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8250096Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8250100Z 2025-12-04T09:42:48.8250181Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8250269Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8250369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8250560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8250634Z res = mod(**inputs) 2025-12-04T09:42:48.8250922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8251002Z outputs = self.bert( 2025-12-04T09:42:48.8251375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8251475Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8251792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8251869Z layer_outputs = layer_module( 2025-12-04T09:42:48.8252112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8252193Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8252467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8252543Z return func(*args, **kwargs) 2025-12-04T09:42:48.8252825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8252932Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8253193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8253262Z return func(*args, **kwargs) 2025-12-04T09:42:48.8253554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8253678Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8253966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8254055Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8254059Z 2025-12-04T09:42:48.8254155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8254366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8254426Z res = mod(**inputs) 2025-12-04T09:42:48.8254701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8254773Z outputs = self.bert( 2025-12-04T09:42:48.8255048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8255115Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8255397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8255463Z layer_outputs = layer_module( 2025-12-04T09:42:48.8255681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8255757Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8255989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8256060Z return func(*args, **kwargs) 2025-12-04T09:42:48.8256333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8256420Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8256664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8256736Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8257046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8257142Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8257415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8257503Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8257507Z 2025-12-04T09:42:48.8257603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8257795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8257855Z res = mod(**inputs) 2025-12-04T09:42:48.8258131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8258204Z outputs = self.bert( 2025-12-04T09:42:48.8258501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8258578Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8258872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8258962Z layer_outputs = layer_module( 2025-12-04T09:42:48.8259181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8259253Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8259478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8259551Z return func(*args, **kwargs) 2025-12-04T09:42:48.8259827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8259910Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8260212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8260312Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8260623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8260724Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8261000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8261107Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8261320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8261398Z return self.act(input) 2025-12-04T09:42:48.8261401Z 2025-12-04T09:42:48.8261497Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8261683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8261754Z res = mod(**inputs) 2025-12-04T09:42:48.8262028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8262097Z outputs = self.bert( 2025-12-04T09:42:48.8262366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8262435Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8262712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8262780Z layer_outputs = layer_module( 2025-12-04T09:42:48.8262998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8263072Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8263300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8263370Z return func(*args, **kwargs) 2025-12-04T09:42:48.8263640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8263718Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8263966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8264035Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8264362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8264491Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8264764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8264871Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8264892Z 2025-12-04T09:42:48.8264992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8265183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8265248Z res = mod(**inputs) 2025-12-04T09:42:48.8265523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8265594Z outputs = self.bert( 2025-12-04T09:42:48.8265866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8265933Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8266211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8266303Z layer_outputs = layer_module( 2025-12-04T09:42:48.8266527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8266603Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8266839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8266912Z return func(*args, **kwargs) 2025-12-04T09:42:48.8267191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8267280Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8267521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8267589Z return func(*args, **kwargs) 2025-12-04T09:42:48.8267895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8267961Z self_outputs = self.self( 2025-12-04T09:42:48.8268192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8268263Z return func(*args, **kwargs) 2025-12-04T09:42:48.8268547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8268631Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8268635Z 2025-12-04T09:42:48.8268734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8268923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8268990Z res = mod(**inputs) 2025-12-04T09:42:48.8269276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8269347Z outputs = self.bert( 2025-12-04T09:42:48.8269625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8269699Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8269985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8270053Z layer_outputs = layer_module( 2025-12-04T09:42:48.8270291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8270376Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8270610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8270711Z return func(*args, **kwargs) 2025-12-04T09:42:48.8271007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8271088Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8271332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8271397Z return func(*args, **kwargs) 2025-12-04T09:42:48.8271688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8271754Z self_outputs = self.self( 2025-12-04T09:42:48.8271988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8272060Z return func(*args, **kwargs) 2025-12-04T09:42:48.8272525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8272646Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8272650Z 2025-12-04T09:42:48.8272760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8272957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8273028Z res = mod(**inputs) 2025-12-04T09:42:48.8273320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8273386Z outputs = self.bert( 2025-12-04T09:42:48.8273686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8273754Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8274046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8274118Z layer_outputs = layer_module( 2025-12-04T09:42:48.8274339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8274420Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8274657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8274722Z return func(*args, **kwargs) 2025-12-04T09:42:48.8275016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8275096Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8275344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8275413Z return func(*args, **kwargs) 2025-12-04T09:42:48.8275699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8275775Z self_outputs = self.self( 2025-12-04T09:42:48.8276014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8276079Z return func(*args, **kwargs) 2025-12-04T09:42:48.8276375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8276452Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8276455Z 2025-12-04T09:42:48.8276576Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8276656Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8276759Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8276980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8277045Z res = mod(**inputs) 2025-12-04T09:42:48.8277353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8277419Z outputs = self.bert( 2025-12-04T09:42:48.8277701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8277777Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8278054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8278124Z layer_outputs = layer_module( 2025-12-04T09:42:48.8278348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8278442Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8278681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8278748Z return func(*args, **kwargs) 2025-12-04T09:42:48.8279025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8279110Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8279343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8279407Z return func(*args, **kwargs) 2025-12-04T09:42:48.8279695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8279818Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8280103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8280186Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8280189Z 2025-12-04T09:42:48.8280288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8280487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8280549Z res = mod(**inputs) 2025-12-04T09:42:48.8280837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8280902Z outputs = self.bert( 2025-12-04T09:42:48.8281178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8281258Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8281540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8281620Z layer_outputs = layer_module( 2025-12-04T09:42:48.8281836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8281911Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8282150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8282217Z return func(*args, **kwargs) 2025-12-04T09:42:48.8282520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8282610Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8282858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8282956Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8283277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8283377Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8283657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8283735Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8283738Z 2025-12-04T09:42:48.8283842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8284029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8284089Z res = mod(**inputs) 2025-12-04T09:42:48.8284372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8284456Z outputs = self.bert( 2025-12-04T09:42:48.8284732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8284808Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8285084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8285158Z layer_outputs = layer_module( 2025-12-04T09:42:48.8285371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8285443Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8285683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8285750Z return func(*args, **kwargs) 2025-12-04T09:42:48.8286033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8286115Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8286369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8286448Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8286759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8286862Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8287151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8287265Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8287484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8287554Z return self.act(input) 2025-12-04T09:42:48.8287558Z 2025-12-04T09:42:48.8287659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8287860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8287923Z res = mod(**inputs) 2025-12-04T09:42:48.8288217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8288285Z outputs = self.bert( 2025-12-04T09:42:48.8288590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8288669Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8288951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8289042Z layer_outputs = layer_module( 2025-12-04T09:42:48.8289307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8289385Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8289628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8289694Z return func(*args, **kwargs) 2025-12-04T09:42:48.8289972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8290063Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8290313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8290391Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8290715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8290842Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8291134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8291218Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8291286Z 2025-12-04T09:42:48.8291452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8291674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8291746Z res = mod(**inputs) 2025-12-04T09:42:48.8292063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8292140Z outputs = self.bert( 2025-12-04T09:42:48.8292447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8292536Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8292839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8292921Z layer_outputs = layer_module( 2025-12-04T09:42:48.8293152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8293238Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8293500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8293573Z return func(*args, **kwargs) 2025-12-04T09:42:48.8293870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8293963Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8294234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8294317Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8294623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8294749Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8295060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.8295137Z return input_tensor + hidden_states 2025-12-04T09:42:48.8295140Z 2025-12-04T09:42:48.8295248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8295464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8295523Z res = mod(**inputs) 2025-12-04T09:42:48.8295819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8295886Z outputs = self.bert( 2025-12-04T09:42:48.8296167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8296242Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8296527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8296603Z layer_outputs = layer_module( 2025-12-04T09:42:48.8296821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8296929Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8297173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8297240Z return func(*args, **kwargs) 2025-12-04T09:42:48.8297526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8297605Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8297850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8297920Z return func(*args, **kwargs) 2025-12-04T09:42:48.8298196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8298262Z self_outputs = self.self( 2025-12-04T09:42:48.8298499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8298567Z return func(*args, **kwargs) 2025-12-04T09:42:48.8298848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8298924Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8298927Z 2025-12-04T09:42:48.8299026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8299224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8299283Z res = mod(**inputs) 2025-12-04T09:42:48.8299576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8299640Z outputs = self.bert( 2025-12-04T09:42:48.8299927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8300005Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8300276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8300341Z layer_outputs = layer_module( 2025-12-04T09:42:48.8300557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8300629Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8300861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8300945Z return func(*args, **kwargs) 2025-12-04T09:42:48.8301221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8301322Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8301551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8301628Z return func(*args, **kwargs) 2025-12-04T09:42:48.8301908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8301972Z self_outputs = self.self( 2025-12-04T09:42:48.8302204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8302267Z return func(*args, **kwargs) 2025-12-04T09:42:48.8302540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8302621Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8302625Z 2025-12-04T09:42:48.8302723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8302936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8302998Z res = mod(**inputs) 2025-12-04T09:42:48.8303270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8303337Z outputs = self.bert( 2025-12-04T09:42:48.8303605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8303671Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8303950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8304017Z layer_outputs = layer_module( 2025-12-04T09:42:48.8304233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8304307Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8304534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8304605Z return func(*args, **kwargs) 2025-12-04T09:42:48.8304874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8304958Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8305183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8305245Z return func(*args, **kwargs) 2025-12-04T09:42:48.8305525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8305591Z self_outputs = self.self( 2025-12-04T09:42:48.8305822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8305891Z return func(*args, **kwargs) 2025-12-04T09:42:48.8306169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8306249Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8306253Z 2025-12-04T09:42:48.8306328Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8306402Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8306505Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8306707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8306770Z res = mod(**inputs) 2025-12-04T09:42:48.8307050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8307155Z outputs = self.bert( 2025-12-04T09:42:48.8307448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8307517Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8307788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8307861Z layer_outputs = layer_module( 2025-12-04T09:42:48.8308070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8308148Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8308376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8308439Z return func(*args, **kwargs) 2025-12-04T09:42:48.8308724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8308823Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8309055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8309126Z return func(*args, **kwargs) 2025-12-04T09:42:48.8309395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8309521Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8309796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8309874Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8309877Z 2025-12-04T09:42:48.8309982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8310169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8310237Z res = mod(**inputs) 2025-12-04T09:42:48.8310510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8310571Z outputs = self.bert( 2025-12-04T09:42:48.8310849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8310917Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8311191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8311263Z layer_outputs = layer_module( 2025-12-04T09:42:48.8311470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8311553Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8311783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8311849Z return func(*args, **kwargs) 2025-12-04T09:42:48.8312137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8312216Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8312466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8312555Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8312858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8312993Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8313286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8313368Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8313378Z 2025-12-04T09:42:48.8313475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8313661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8313729Z res = mod(**inputs) 2025-12-04T09:42:48.8314006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8314070Z outputs = self.bert( 2025-12-04T09:42:48.8314352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8314421Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8314722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8314789Z layer_outputs = layer_module( 2025-12-04T09:42:48.8314999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8315083Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8315310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8315375Z return func(*args, **kwargs) 2025-12-04T09:42:48.8315661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8315740Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8315990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8316066Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8316367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8316472Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8316740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8316852Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8317058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8317125Z return self.act(input) 2025-12-04T09:42:48.8317128Z 2025-12-04T09:42:48.8317232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8317426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8317491Z res = mod(**inputs) 2025-12-04T09:42:48.8317792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8317859Z outputs = self.bert( 2025-12-04T09:42:48.8318148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8318219Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8318509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8318614Z layer_outputs = layer_module( 2025-12-04T09:42:48.8318852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8318960Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8319219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8319309Z return func(*args, **kwargs) 2025-12-04T09:42:48.8319597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8319678Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8319935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8320016Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8320335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8320478Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8320784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8320864Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8320868Z 2025-12-04T09:42:48.8320979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8321180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8321248Z res = mod(**inputs) 2025-12-04T09:42:48.8321538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8321603Z outputs = self.bert( 2025-12-04T09:42:48.8321894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8321964Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8322249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8322326Z layer_outputs = layer_module( 2025-12-04T09:42:48.8322546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8322627Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8322866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8322932Z return func(*args, **kwargs) 2025-12-04T09:42:48.8323224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8323304Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8323548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8323618Z return func(*args, **kwargs) 2025-12-04T09:42:48.8323904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8323977Z self_outputs = self.self( 2025-12-04T09:42:48.8324217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8324282Z return func(*args, **kwargs) 2025-12-04T09:42:48.8324574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8324654Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8324676Z 2025-12-04T09:42:48.8324789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8324989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8325074Z res = mod(**inputs) 2025-12-04T09:42:48.8325388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8325456Z outputs = self.bert( 2025-12-04T09:42:48.8325755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8325825Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8326117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8326193Z layer_outputs = layer_module( 2025-12-04T09:42:48.8326419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8326495Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8326745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8326835Z return func(*args, **kwargs) 2025-12-04T09:42:48.8327173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8327254Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8327496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8327571Z return func(*args, **kwargs) 2025-12-04T09:42:48.8327859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8327937Z self_outputs = self.self( 2025-12-04T09:42:48.8328176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8328244Z return func(*args, **kwargs) 2025-12-04T09:42:48.8328542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8328618Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8328621Z 2025-12-04T09:42:48.8328723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8328929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8328992Z res = mod(**inputs) 2025-12-04T09:42:48.8329287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8329354Z outputs = self.bert( 2025-12-04T09:42:48.8329643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8329723Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8330014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8330091Z layer_outputs = layer_module( 2025-12-04T09:42:48.8330325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8330407Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8330675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8330745Z return func(*args, **kwargs) 2025-12-04T09:42:48.8331066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8331162Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8331500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8331626Z return func(*args, **kwargs) 2025-12-04T09:42:48.8331954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8332032Z self_outputs = self.self( 2025-12-04T09:42:48.8332298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8332371Z return func(*args, **kwargs) 2025-12-04T09:42:48.8332684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8332779Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8332783Z 2025-12-04T09:42:48.8332869Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8332960Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8333073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8333297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8333383Z res = mod(**inputs) 2025-12-04T09:42:48.8333673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8333746Z outputs = self.bert( 2025-12-04T09:42:48.8334032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8334103Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8334396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8334464Z layer_outputs = layer_module( 2025-12-04T09:42:48.8334685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8334770Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8335011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8335086Z return func(*args, **kwargs) 2025-12-04T09:42:48.8335370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8335447Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8335693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8335760Z return func(*args, **kwargs) 2025-12-04T09:42:48.8336049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8336182Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8336467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8336555Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8336559Z 2025-12-04T09:42:48.8336659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8336851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8336921Z res = mod(**inputs) 2025-12-04T09:42:48.8337208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8337301Z outputs = self.bert( 2025-12-04T09:42:48.8337593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8337664Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8337977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8338061Z layer_outputs = layer_module( 2025-12-04T09:42:48.8338292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8338369Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8338613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8338697Z return func(*args, **kwargs) 2025-12-04T09:42:48.8338977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8339058Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8339316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8339409Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8339727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8339828Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8340104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8340191Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8340195Z 2025-12-04T09:42:48.8340291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8340490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8340551Z res = mod(**inputs) 2025-12-04T09:42:48.8340829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8340903Z outputs = self.bert( 2025-12-04T09:42:48.8341184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8341254Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8341542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8341608Z layer_outputs = layer_module( 2025-12-04T09:42:48.8341838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8341916Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8342157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8342233Z return func(*args, **kwargs) 2025-12-04T09:42:48.8342537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8342627Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8342891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8342966Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8343289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8343391Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8343695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8343815Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8344056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8344149Z return self.act(input) 2025-12-04T09:42:48.8344152Z 2025-12-04T09:42:48.8344257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8344452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8344524Z res = mod(**inputs) 2025-12-04T09:42:48.8344813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8344885Z outputs = self.bert( 2025-12-04T09:42:48.8345173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8345244Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8345545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8345635Z layer_outputs = layer_module( 2025-12-04T09:42:48.8345852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8345935Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8346170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8346243Z return func(*args, **kwargs) 2025-12-04T09:42:48.8346523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8346604Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8346865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8346945Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8347261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8347392Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8347668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8347754Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8347759Z 2025-12-04T09:42:48.8347858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8348053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8348121Z res = mod(**inputs) 2025-12-04T09:42:48.8348403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8348478Z outputs = self.bert( 2025-12-04T09:42:48.8348773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8348844Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8349131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8349198Z layer_outputs = layer_module( 2025-12-04T09:42:48.8349423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8349499Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8349753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8349829Z return func(*args, **kwargs) 2025-12-04T09:42:48.8350123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8350220Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8350483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8350555Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8350871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8350998Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8351277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.8351362Z return input_tensor + hidden_states 2025-12-04T09:42:48.8351365Z 2025-12-04T09:42:48.8351490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8351698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8351765Z res = mod(**inputs) 2025-12-04T09:42:48.8352070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8352141Z outputs = self.bert( 2025-12-04T09:42:48.8352424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8352493Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8352791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8352862Z layer_outputs = layer_module( 2025-12-04T09:42:48.8353091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8353170Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8353420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8353495Z return func(*args, **kwargs) 2025-12-04T09:42:48.8353798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8353881Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8354118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8354184Z return func(*args, **kwargs) 2025-12-04T09:42:48.8354478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8354546Z self_outputs = self.self( 2025-12-04T09:42:48.8354784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8354855Z return func(*args, **kwargs) 2025-12-04T09:42:48.8355139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8355223Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8355226Z 2025-12-04T09:42:48.8355325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8355517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8355584Z res = mod(**inputs) 2025-12-04T09:42:48.8355894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8355962Z outputs = self.bert( 2025-12-04T09:42:48.8356262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8356347Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8356635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8356704Z layer_outputs = layer_module( 2025-12-04T09:42:48.8356921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8357003Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8357239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8357393Z return func(*args, **kwargs) 2025-12-04T09:42:48.8357698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8357843Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8358135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8358226Z return func(*args, **kwargs) 2025-12-04T09:42:48.8358780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8358906Z self_outputs = self.self( 2025-12-04T09:42:48.8359199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8359289Z return func(*args, **kwargs) 2025-12-04T09:42:48.8359595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8359713Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8359720Z 2025-12-04T09:42:48.8359876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8360148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8360221Z res = mod(**inputs) 2025-12-04T09:42:48.8379186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8379359Z outputs = self.bert( 2025-12-04T09:42:48.8379731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8379817Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8380143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8380221Z layer_outputs = layer_module( 2025-12-04T09:42:48.8380459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8380556Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8380815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8380896Z return func(*args, **kwargs) 2025-12-04T09:42:48.8381193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8381282Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8381539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8381724Z return func(*args, **kwargs) 2025-12-04T09:42:48.8382020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8382143Z self_outputs = self.self( 2025-12-04T09:42:48.8382422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8382502Z return func(*args, **kwargs) 2025-12-04T09:42:48.8382796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8382878Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8382886Z 2025-12-04T09:42:48.8382984Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8383064Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8383175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8383394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8383462Z res = mod(**inputs) 2025-12-04T09:42:48.8383771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8383879Z outputs = self.bert( 2025-12-04T09:42:48.8384200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8384294Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8384608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8384691Z layer_outputs = layer_module( 2025-12-04T09:42:48.8384933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8385017Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8385285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8385361Z return func(*args, **kwargs) 2025-12-04T09:42:48.8385679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8385778Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8386042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8386121Z return func(*args, **kwargs) 2025-12-04T09:42:48.8386433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8386570Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8386886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8386976Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8386984Z 2025-12-04T09:42:48.8387107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8387325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8387394Z res = mod(**inputs) 2025-12-04T09:42:48.8387761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8387839Z outputs = self.bert( 2025-12-04T09:42:48.8388155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8388242Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8388606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8388692Z layer_outputs = layer_module( 2025-12-04T09:42:48.8388956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8389044Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8389354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8389430Z return func(*args, **kwargs) 2025-12-04T09:42:48.8389748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8389848Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8390138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8390229Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8390571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8390706Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8391028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8391115Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8391119Z 2025-12-04T09:42:48.8391239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8391467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8391534Z res = mod(**inputs) 2025-12-04T09:42:48.8391848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8391913Z outputs = self.bert( 2025-12-04T09:42:48.8392224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8392304Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8392620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8392703Z layer_outputs = layer_module( 2025-12-04T09:42:48.8392938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8393021Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8393282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8393354Z return func(*args, **kwargs) 2025-12-04T09:42:48.8393667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8393754Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8394036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8394124Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8394461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8394580Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8394886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8395003Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8395257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8395335Z return self.act(input) 2025-12-04T09:42:48.8395339Z 2025-12-04T09:42:48.8395451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8395689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8395775Z res = mod(**inputs) 2025-12-04T09:42:48.8396094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8396163Z outputs = self.bert( 2025-12-04T09:42:48.8396467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8396553Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8396861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8396943Z layer_outputs = layer_module( 2025-12-04T09:42:48.8397179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8397282Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8397554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8397628Z return func(*args, **kwargs) 2025-12-04T09:42:48.8397938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8398031Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8398312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8398403Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8398748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8398895Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8399215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8399300Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8399304Z 2025-12-04T09:42:48.8399420Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8399633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8399699Z res = mod(**inputs) 2025-12-04T09:42:48.8400020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8400093Z outputs = self.bert( 2025-12-04T09:42:48.8400402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8400492Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8400811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8400892Z layer_outputs = layer_module( 2025-12-04T09:42:48.8401131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8401212Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8401483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8401556Z return func(*args, **kwargs) 2025-12-04T09:42:48.8401890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8401981Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8402234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8402338Z return func(*args, **kwargs) 2025-12-04T09:42:48.8402661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8402738Z self_outputs = self.self( 2025-12-04T09:42:48.8403001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8403071Z return func(*args, **kwargs) 2025-12-04T09:42:48.8403382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T09:42:48.8403468Z query_layer = self.query(hidden_states) 2025-12-04T09:42:48.8403473Z 2025-12-04T09:42:48.8403580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8403798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8403887Z res = mod(**inputs) 2025-12-04T09:42:48.8404203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8404272Z outputs = self.bert( 2025-12-04T09:42:48.8404573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8404658Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8404961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8405036Z layer_outputs = layer_module( 2025-12-04T09:42:48.8405279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8405360Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8405624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8405696Z return func(*args, **kwargs) 2025-12-04T09:42:48.8405998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8406091Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8406343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8406413Z return func(*args, **kwargs) 2025-12-04T09:42:48.8406726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8406798Z self_outputs = self.self( 2025-12-04T09:42:48.8407058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8407131Z return func(*args, **kwargs) 2025-12-04T09:42:48.8407438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T09:42:48.8407528Z key_layer = self.key(current_states) 2025-12-04T09:42:48.8407532Z 2025-12-04T09:42:48.8407640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8407853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8407920Z res = mod(**inputs) 2025-12-04T09:42:48.8408245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8408325Z outputs = self.bert( 2025-12-04T09:42:48.8408629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8408738Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8409075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8409159Z layer_outputs = layer_module( 2025-12-04T09:42:48.8409395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8409477Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8409743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8409815Z return func(*args, **kwargs) 2025-12-04T09:42:48.8410121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8410213Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8410466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8410565Z return func(*args, **kwargs) 2025-12-04T09:42:48.8410883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T09:42:48.8410956Z self_outputs = self.self( 2025-12-04T09:42:48.8411308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8411391Z return func(*args, **kwargs) 2025-12-04T09:42:48.8411711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T09:42:48.8411797Z value_layer = self.value(current_states) 2025-12-04T09:42:48.8411801Z 2025-12-04T09:42:48.8411889Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8411984Z cudagraph partition due to non gpu ops 2025-12-04T09:42:48.8412097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8412306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8412386Z res = mod(**inputs) 2025-12-04T09:42:48.8412693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8412772Z outputs = self.bert( 2025-12-04T09:42:48.8413073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8413149Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8413460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8413536Z layer_outputs = layer_module( 2025-12-04T09:42:48.8413770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8413861Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8414123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8414200Z return func(*args, **kwargs) 2025-12-04T09:42:48.8414512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T09:42:48.8414595Z self_attention_outputs = self.attention( 2025-12-04T09:42:48.8414857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8414947Z return func(*args, **kwargs) 2025-12-04T09:42:48.8415273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T09:42:48.8415443Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:42:48.8415779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T09:42:48.8415876Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8415880Z 2025-12-04T09:42:48.8415989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8416198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8416273Z res = mod(**inputs) 2025-12-04T09:42:48.8416581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8416655Z outputs = self.bert( 2025-12-04T09:42:48.8416959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8417057Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8417366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8417440Z layer_outputs = layer_module( 2025-12-04T09:42:48.8417687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8417764Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8418003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8418076Z return func(*args, **kwargs) 2025-12-04T09:42:48.8418365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8418448Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8418716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8418794Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8419118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8419223Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8419505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T09:42:48.8419593Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8419596Z 2025-12-04T09:42:48.8419706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8419923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8419989Z res = mod(**inputs) 2025-12-04T09:42:48.8420296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8420373Z outputs = self.bert( 2025-12-04T09:42:48.8420672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8420749Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8421064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8421135Z layer_outputs = layer_module( 2025-12-04T09:42:48.8421388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8421466Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8421711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8421804Z return func(*args, **kwargs) 2025-12-04T09:42:48.8422108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8422200Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8422462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8422538Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8422864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T09:42:48.8422968Z intermediate_output = self.intermediate(ln_output) 2025-12-04T09:42:48.8423257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T09:42:48.8423396Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:42:48.8423618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:42:48.8423696Z return self.act(input) 2025-12-04T09:42:48.8423699Z 2025-12-04T09:42:48.8423804Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8424003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8424074Z res = mod(**inputs) 2025-12-04T09:42:48.8424371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8424444Z outputs = self.bert( 2025-12-04T09:42:48.8424739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8424811Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8425114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8425186Z layer_outputs = layer_module( 2025-12-04T09:42:48.8425410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8425495Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8425740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8425818Z return func(*args, **kwargs) 2025-12-04T09:42:48.8426112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8426195Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8426469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8426547Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8426874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8427009Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8427296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T09:42:48.8427385Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8427389Z 2025-12-04T09:42:48.8427491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8427708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8427781Z res = mod(**inputs) 2025-12-04T09:42:48.8428072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T09:42:48.8428177Z outputs = self.bert( 2025-12-04T09:42:48.8428481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T09:42:48.8428555Z encoder_outputs = self.encoder( 2025-12-04T09:42:48.8428850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T09:42:48.8428920Z layer_outputs = layer_module( 2025-12-04T09:42:48.8429149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:42:48.8429229Z return super().__call__(*args, **kwargs) 2025-12-04T09:42:48.8429471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:42:48.8429549Z return func(*args, **kwargs) 2025-12-04T09:42:48.8429856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T09:42:48.8429939Z layer_output = apply_chunking_to_forward( 2025-12-04T09:42:48.8430209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:42:48.8430284Z return forward_fn(*input_tensors) 2025-12-04T09:42:48.8430615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T09:42:48.8430748Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:42:48.8431030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T09:42:48.8431113Z return input_tensor + hidden_states 2025-12-04T09:42:48.8431120Z 2025-12-04T09:42:48.8431220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8431421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8431483Z res = mod(**inputs) 2025-12-04T09:42:48.8431767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-12-04T09:42:48.8431868Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:42:48.8432149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-12-04T09:42:48.8432270Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:42:48.8432556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 632, in forward 2025-12-04T09:42:48.8432649Z hidden_states = self.transform(hidden_states) 2025-12-04T09:42:48.8432940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 607, in forward 2025-12-04T09:42:48.8433018Z hidden_states = self.dense(hidden_states) 2025-12-04T09:42:48.8433021Z 2025-12-04T09:42:48.8433123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8433329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8433390Z res = mod(**inputs) 2025-12-04T09:42:48.8433701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-12-04T09:42:48.8433819Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:42:48.8434098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-12-04T09:42:48.8434232Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:42:48.8434524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 633, in forward 2025-12-04T09:42:48.8434621Z hidden_states = self.decoder(hidden_states) 2025-12-04T09:42:48.8434624Z 2025-12-04T09:42:48.8434721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:42:48.8434913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:42:48.8434983Z res = mod(**inputs) 2025-12-04T09:42:48.8435285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1078, in forward 2025-12-04T09:42:48.8435359Z lm_loss = self.loss_function( 2025-12-04T09:42:48.8435617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:42:48.8435814Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:42:48.8436078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:42:48.8436272Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:42:48.8436276Z 2025-12-04T09:43:00.4639815Z Compilation time (from dynamo_timed): 25.004073441 2025-12-04T09:43:00.4671534Z pass 2025-12-04T09:43:00.4672965Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:00.4676269Z TIMING: _recursive_pre_grad_passes:0.01042 _recursive_joint_graph_passes:0.75956 _recursive_post_grad_passes:0.11888 async_compile.wait:0.85303 code_gen:11.26153 inductor_compile:13.68181 backend_compile:19.58077 gc:0.00023 entire_frame_compile:25.00407 total_wall_time:25.00407 2025-12-04T09:43:00.4679319Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:16128 | FakeTensor.__torch_dispatch__:8250 | ProxyTorchDispatchMode.__torch_dispatch__:4916 2025-12-04T09:43:00.4679914Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-12-04T09:43:03.4299164Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:43:03.4300342Z import pynvml # type: ignore[import] 2025-12-04T09:43:06.8501328Z 2025-12-04T09:43:07.3453897Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:43:07.3454181Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:43:07.3520819Z cpu eval MobileBertForMaskedLM 2025-12-04T09:43:07.6218670Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:07.7871991Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:08.1361442Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:34.0333885Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0337007Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0337342Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0337839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0338207Z res = mod(**inputs) 2025-12-04T09:43:34.0341425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0342532Z outputs = self.mobilebert( 2025-12-04T09:43:34.0343134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T09:43:34.0343820Z embedding_output = self.embeddings( 2025-12-04T09:43:34.0344997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-12-04T09:43:34.0345590Z inputs_embeds = torch.cat( 2025-12-04T09:43:34.0345723Z 2025-12-04T09:43:34.0345842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0346222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0346562Z res = mod(**inputs) 2025-12-04T09:43:34.0346973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T09:43:34.0347431Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:43:34.0347880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T09:43:34.0348351Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:43:34.0348886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-12-04T09:43:34.0349432Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-12-04T09:43:34.0349695Z 2025-12-04T09:43:34.0349802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0350174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0350487Z res = mod(**inputs) 2025-12-04T09:43:34.0350879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0351300Z outputs = self.mobilebert( 2025-12-04T09:43:34.0351701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T09:43:34.0352118Z embedding_output = self.embeddings( 2025-12-04T09:43:34.0352534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-12-04T09:43:34.0353000Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-12-04T09:43:34.0353175Z 2025-12-04T09:43:34.0353288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0353642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0353963Z res = mod(**inputs) 2025-12-04T09:43:34.0354353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0354758Z outputs = self.mobilebert( 2025-12-04T09:43:34.0355155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T09:43:34.0355585Z embedding_output = self.embeddings( 2025-12-04T09:43:34.0356001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-12-04T09:43:34.0356418Z embeddings = self.LayerNorm(embeddings) 2025-12-04T09:43:34.0356837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0357273Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0357418Z 2025-12-04T09:43:34.0357528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0357920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0358245Z res = mod(**inputs) 2025-12-04T09:43:34.0358627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0359225Z outputs = self.mobilebert( 2025-12-04T09:43:34.0359650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0360065Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0360474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0360897Z layer_outputs = layer_module( 2025-12-04T09:43:34.0361285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0361800Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0362291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0362717Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0363150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0363552Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0363688Z 2025-12-04T09:43:34.0363789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0364145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0364460Z res = mod(**inputs) 2025-12-04T09:43:34.0364839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0365257Z outputs = self.mobilebert( 2025-12-04T09:43:34.0365651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0366087Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0366519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0366953Z layer_outputs = layer_module( 2025-12-04T09:43:34.0367372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0367820Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0368260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0368687Z self_outputs = self.self( 2025-12-04T09:43:34.0369094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.0369529Z self.value(value_tensor) 2025-12-04T09:43:34.0369651Z 2025-12-04T09:43:34.0369771Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0370137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0370471Z res = mod(**inputs) 2025-12-04T09:43:34.0370874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0371583Z outputs = self.mobilebert( 2025-12-04T09:43:34.0372012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0372747Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0373243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0373678Z layer_outputs = layer_module( 2025-12-04T09:43:34.0374101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0374673Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0375241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.0375714Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.0376186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0376635Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0376778Z 2025-12-04T09:43:34.0376893Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0377262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0377587Z res = mod(**inputs) 2025-12-04T09:43:34.0377971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0378460Z outputs = self.mobilebert( 2025-12-04T09:43:34.0378882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0379331Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0379766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0380217Z layer_outputs = layer_module( 2025-12-04T09:43:34.0380646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0381181Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0381715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0382196Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0382680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.0383139Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.0383592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0384046Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0384211Z 2025-12-04T09:43:34.0384317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0384703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0385045Z res = mod(**inputs) 2025-12-04T09:43:34.0385450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0385892Z outputs = self.mobilebert( 2025-12-04T09:43:34.0386314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0386745Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0387177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0387613Z layer_outputs = layer_module( 2025-12-04T09:43:34.0388046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0388514Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0388967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0389392Z self_outputs = self.self( 2025-12-04T09:43:34.0389835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.0390280Z self.query(query_tensor) 2025-12-04T09:43:34.0390411Z 2025-12-04T09:43:34.0390521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0390903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0391240Z res = mod(**inputs) 2025-12-04T09:43:34.0391657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0392105Z outputs = self.mobilebert( 2025-12-04T09:43:34.0392537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0392982Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0393438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0393897Z layer_outputs = layer_module( 2025-12-04T09:43:34.0394342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0394800Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0395261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0395712Z self_outputs = self.self( 2025-12-04T09:43:34.0396141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.0396580Z self.key(key_tensor) 2025-12-04T09:43:34.0396697Z 2025-12-04T09:43:34.0396793Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0397027Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0397282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0397671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0398014Z res = mod(**inputs) 2025-12-04T09:43:34.0398419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0398864Z outputs = self.mobilebert( 2025-12-04T09:43:34.0399296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0399737Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0400169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0400621Z layer_outputs = layer_module( 2025-12-04T09:43:34.0401057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0401510Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0401968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0402461Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0402953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.0403401Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0403560Z 2025-12-04T09:43:34.0403697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0404099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0404449Z res = mod(**inputs) 2025-12-04T09:43:34.0404893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0405344Z outputs = self.mobilebert( 2025-12-04T09:43:34.0405791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0406239Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0406683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0407124Z layer_outputs = layer_module( 2025-12-04T09:43:34.0407562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0407996Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0408439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0408951Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0409436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.0409921Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0410409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0410862Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0411030Z 2025-12-04T09:43:34.0411147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0411655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0412010Z res = mod(**inputs) 2025-12-04T09:43:34.0412505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0412945Z outputs = self.mobilebert( 2025-12-04T09:43:34.0413379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0413832Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0414268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0414697Z layer_outputs = layer_module( 2025-12-04T09:43:34.0415152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0415619Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0416080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0416559Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0417040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0417495Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0417644Z 2025-12-04T09:43:34.0417753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0418136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0418475Z res = mod(**inputs) 2025-12-04T09:43:34.0418878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0419337Z outputs = self.mobilebert( 2025-12-04T09:43:34.0419758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0420212Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0420640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0421094Z layer_outputs = layer_module( 2025-12-04T09:43:34.0421606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0422111Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0422567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0423059Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0423552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0424051Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0424255Z 2025-12-04T09:43:34.0424365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0424749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0425099Z res = mod(**inputs) 2025-12-04T09:43:34.0425507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0425936Z outputs = self.mobilebert( 2025-12-04T09:43:34.0426358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0426794Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0427227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0427655Z layer_outputs = layer_module( 2025-12-04T09:43:34.0428083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0428543Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0428996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0429492Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0429981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0430428Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0430573Z 2025-12-04T09:43:34.0430683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0431052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0431388Z res = mod(**inputs) 2025-12-04T09:43:34.0431794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0432221Z outputs = self.mobilebert( 2025-12-04T09:43:34.0432647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0433085Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0433503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0433937Z layer_outputs = layer_module( 2025-12-04T09:43:34.0434382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0434836Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0435283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0435791Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0436311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0436800Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0437283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0437737Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0437893Z 2025-12-04T09:43:34.0438007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0438459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0438791Z res = mod(**inputs) 2025-12-04T09:43:34.0439206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0439669Z outputs = self.mobilebert( 2025-12-04T09:43:34.0440086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0440526Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0440957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0441395Z layer_outputs = layer_module( 2025-12-04T09:43:34.0441821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0442291Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0442757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0443244Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0443735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0444200Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0444348Z 2025-12-04T09:43:34.0444468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0444878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0445245Z res = mod(**inputs) 2025-12-04T09:43:34.0445670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0446180Z outputs = self.mobilebert( 2025-12-04T09:43:34.0446628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0447094Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0447542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0448002Z layer_outputs = layer_module( 2025-12-04T09:43:34.0448445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0448924Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0449406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0449937Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0450434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0450940Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0451146Z 2025-12-04T09:43:34.0451323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0451744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0452116Z res = mod(**inputs) 2025-12-04T09:43:34.0452534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0452978Z outputs = self.mobilebert( 2025-12-04T09:43:34.0453417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0453873Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0454318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0454758Z layer_outputs = layer_module( 2025-12-04T09:43:34.0455220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0455704Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0456188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0456694Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0457205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0457682Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0457829Z 2025-12-04T09:43:34.0457952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0458326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0458679Z res = mod(**inputs) 2025-12-04T09:43:34.0459097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0459551Z outputs = self.mobilebert( 2025-12-04T09:43:34.0459995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0460455Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0460892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0461333Z layer_outputs = layer_module( 2025-12-04T09:43:34.0461779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0462256Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0462733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0463250Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0463746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0464230Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0464710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0465171Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0465342Z 2025-12-04T09:43:34.0465489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0465862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0466206Z res = mod(**inputs) 2025-12-04T09:43:34.0466663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0467107Z outputs = self.mobilebert( 2025-12-04T09:43:34.0467553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0467986Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0468420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0468859Z layer_outputs = layer_module( 2025-12-04T09:43:34.0469284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0469750Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0470215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0470729Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0471216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0471680Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0471836Z 2025-12-04T09:43:34.0471946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0472552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0472899Z res = mod(**inputs) 2025-12-04T09:43:34.0473318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0473757Z outputs = self.mobilebert( 2025-12-04T09:43:34.0474173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0474613Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0475043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0475477Z layer_outputs = layer_module( 2025-12-04T09:43:34.0475901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0476361Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0476822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0477299Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0477769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0478251Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0478426Z 2025-12-04T09:43:34.0478542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0478917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0479250Z res = mod(**inputs) 2025-12-04T09:43:34.0479656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0480092Z outputs = self.mobilebert( 2025-12-04T09:43:34.0480502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0480999Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0481428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0481904Z layer_outputs = layer_module( 2025-12-04T09:43:34.0482339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0482828Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0483291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0483789Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0484285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0484742Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0484893Z 2025-12-04T09:43:34.0485012Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0485386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0485764Z res = mod(**inputs) 2025-12-04T09:43:34.0486173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0486610Z outputs = self.mobilebert( 2025-12-04T09:43:34.0487026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0487464Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0487893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0488329Z layer_outputs = layer_module( 2025-12-04T09:43:34.0488760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0489226Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0489684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0490183Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0490682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0491168Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0491761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0492239Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0492410Z 2025-12-04T09:43:34.0492525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0492964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0493309Z res = mod(**inputs) 2025-12-04T09:43:34.0493730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0494198Z outputs = self.mobilebert( 2025-12-04T09:43:34.0494639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0495099Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0495555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0496019Z layer_outputs = layer_module( 2025-12-04T09:43:34.0496524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0497032Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0497540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0498049Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0498199Z 2025-12-04T09:43:34.0498337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0498732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0499109Z res = mod(**inputs) 2025-12-04T09:43:34.0499538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0499990Z outputs = self.mobilebert( 2025-12-04T09:43:34.0500435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0500888Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0501336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0501793Z layer_outputs = layer_module( 2025-12-04T09:43:34.0502207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0502674Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0503134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0503616Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0503799Z 2025-12-04T09:43:34.0503907Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0504294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0504645Z res = mod(**inputs) 2025-12-04T09:43:34.0505059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0505502Z outputs = self.mobilebert( 2025-12-04T09:43:34.0505928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0506335Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0506737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0507164Z layer_outputs = layer_module( 2025-12-04T09:43:34.0507587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0508109Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0508620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.0509076Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.0509235Z 2025-12-04T09:43:34.0509349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0509718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0510057Z res = mod(**inputs) 2025-12-04T09:43:34.0510467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0510891Z outputs = self.mobilebert( 2025-12-04T09:43:34.0511360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0511799Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0512207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0512644Z layer_outputs = layer_module( 2025-12-04T09:43:34.0513110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0513641Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0514167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.0514645Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.0515130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0515605Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0515754Z 2025-12-04T09:43:34.0515877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0516233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0516588Z res = mod(**inputs) 2025-12-04T09:43:34.0516976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0517398Z outputs = self.mobilebert( 2025-12-04T09:43:34.0517822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0518261Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0518688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0519134Z layer_outputs = layer_module( 2025-12-04T09:43:34.0519564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0520094Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0520632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0521110Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0521602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.0522051Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0522197Z 2025-12-04T09:43:34.0522302Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0522688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0523026Z res = mod(**inputs) 2025-12-04T09:43:34.0523434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0523873Z outputs = self.mobilebert( 2025-12-04T09:43:34.0524295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0524741Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0525171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0525604Z layer_outputs = layer_module( 2025-12-04T09:43:34.0526033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0526578Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0527096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0527582Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0528104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.0528593Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0529073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0529529Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0529690Z 2025-12-04T09:43:34.0529796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0530180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0530508Z res = mod(**inputs) 2025-12-04T09:43:34.0530914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0531456Z outputs = self.mobilebert( 2025-12-04T09:43:34.0531907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0532350Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0532805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0533257Z layer_outputs = layer_module( 2025-12-04T09:43:34.0533673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0534210Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0534740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0535226Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0535698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0536148Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0536300Z 2025-12-04T09:43:34.0536408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0536777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0537116Z res = mod(**inputs) 2025-12-04T09:43:34.0537521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0537959Z outputs = self.mobilebert( 2025-12-04T09:43:34.0538377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0538823Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0539248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0539682Z layer_outputs = layer_module( 2025-12-04T09:43:34.0540096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0540545Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0540994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0541422Z self_outputs = self.self( 2025-12-04T09:43:34.0541864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.0542300Z self.value(value_tensor) 2025-12-04T09:43:34.0542419Z 2025-12-04T09:43:34.0542537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0542931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0543267Z res = mod(**inputs) 2025-12-04T09:43:34.0543689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0544124Z outputs = self.mobilebert( 2025-12-04T09:43:34.0544537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0544974Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0545412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0545850Z layer_outputs = layer_module( 2025-12-04T09:43:34.0546270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0546817Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0547345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.0547821Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.0548286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0548733Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0548876Z 2025-12-04T09:43:34.0548990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0549360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0549712Z res = mod(**inputs) 2025-12-04T09:43:34.0550120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0550559Z outputs = self.mobilebert( 2025-12-04T09:43:34.0550977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0551408Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0551838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0552269Z layer_outputs = layer_module( 2025-12-04T09:43:34.0552688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0553212Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0553738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0554211Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0554678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.0555124Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.0555564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0556011Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0556170Z 2025-12-04T09:43:34.0556275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0556677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0557013Z res = mod(**inputs) 2025-12-04T09:43:34.0557408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0557914Z outputs = self.mobilebert( 2025-12-04T09:43:34.0558352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0558782Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0559208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0559646Z layer_outputs = layer_module( 2025-12-04T09:43:34.0560074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0560526Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0560976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0561413Z self_outputs = self.self( 2025-12-04T09:43:34.0561858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.0562296Z self.query(query_tensor) 2025-12-04T09:43:34.0562428Z 2025-12-04T09:43:34.0562536Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0562916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0563247Z res = mod(**inputs) 2025-12-04T09:43:34.0563647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0564079Z outputs = self.mobilebert( 2025-12-04T09:43:34.0564502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0564930Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0565367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0565807Z layer_outputs = layer_module( 2025-12-04T09:43:34.0566237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0566682Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0567130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0567565Z self_outputs = self.self( 2025-12-04T09:43:34.0567984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.0568417Z self.key(key_tensor) 2025-12-04T09:43:34.0568539Z 2025-12-04T09:43:34.0568624Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0568859Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0569103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0569484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0569822Z res = mod(**inputs) 2025-12-04T09:43:34.0570222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0570657Z outputs = self.mobilebert( 2025-12-04T09:43:34.0571091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0571643Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0572123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0572741Z layer_outputs = layer_module( 2025-12-04T09:43:34.0573187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0573711Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0574198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0574708Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0575209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.0575677Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0575838Z 2025-12-04T09:43:34.0575951Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0576341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0576693Z res = mod(**inputs) 2025-12-04T09:43:34.0577105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0577595Z outputs = self.mobilebert( 2025-12-04T09:43:34.0578038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0578498Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0578938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0579397Z layer_outputs = layer_module( 2025-12-04T09:43:34.0579843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0580306Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0580779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0581293Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0581808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.0582316Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0582827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0583324Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0583488Z 2025-12-04T09:43:34.0583613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0584003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0584363Z res = mod(**inputs) 2025-12-04T09:43:34.0584777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0585216Z outputs = self.mobilebert( 2025-12-04T09:43:34.0585648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0586090Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0586526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0586960Z layer_outputs = layer_module( 2025-12-04T09:43:34.0587391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0587891Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0588351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0588847Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0589346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0589773Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0589910Z 2025-12-04T09:43:34.0590016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0590364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0590699Z res = mod(**inputs) 2025-12-04T09:43:34.0591098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0591535Z outputs = self.mobilebert( 2025-12-04T09:43:34.0591955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0592390Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0592837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0593265Z layer_outputs = layer_module( 2025-12-04T09:43:34.0593695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0594153Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0594603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0595083Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0595561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0596017Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0596188Z 2025-12-04T09:43:34.0596290Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0596661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0596999Z res = mod(**inputs) 2025-12-04T09:43:34.0597409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0597833Z outputs = self.mobilebert( 2025-12-04T09:43:34.0598250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0598687Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0599139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0599568Z layer_outputs = layer_module( 2025-12-04T09:43:34.0600003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0600467Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0600918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0601420Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0601907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0602367Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0602512Z 2025-12-04T09:43:34.0602640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0603016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0603355Z res = mod(**inputs) 2025-12-04T09:43:34.0603779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0604204Z outputs = self.mobilebert( 2025-12-04T09:43:34.0604639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0605080Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0605500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0605939Z layer_outputs = layer_module( 2025-12-04T09:43:34.0606369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0606826Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0607274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0607791Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0608277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0608753Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0609225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0609678Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0609830Z 2025-12-04T09:43:34.0609941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0610318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0610643Z res = mod(**inputs) 2025-12-04T09:43:34.0611055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0611594Z outputs = self.mobilebert( 2025-12-04T09:43:34.0612028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0612489Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0612941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0613380Z layer_outputs = layer_module( 2025-12-04T09:43:34.0613800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0614262Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0614719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0615195Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0615673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0616122Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0616269Z 2025-12-04T09:43:34.0616382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0616756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0617090Z res = mod(**inputs) 2025-12-04T09:43:34.0617486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0617891Z outputs = self.mobilebert( 2025-12-04T09:43:34.0618278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0618694Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0619106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0619499Z layer_outputs = layer_module( 2025-12-04T09:43:34.0619891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0620310Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0620733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0621167Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0621601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0622041Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0622227Z 2025-12-04T09:43:34.0622333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0622675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0622992Z res = mod(**inputs) 2025-12-04T09:43:34.0623366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0623763Z outputs = self.mobilebert( 2025-12-04T09:43:34.0624148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0624550Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0624947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0625341Z layer_outputs = layer_module( 2025-12-04T09:43:34.0625737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0626161Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0626580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0627022Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0627474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0627892Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0628023Z 2025-12-04T09:43:34.0628129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0628467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0628782Z res = mod(**inputs) 2025-12-04T09:43:34.0629159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0629558Z outputs = self.mobilebert( 2025-12-04T09:43:34.0629951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0630359Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0630756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0631153Z layer_outputs = layer_module( 2025-12-04T09:43:34.0631565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0631994Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0632420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0632897Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0633364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0633811Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0634248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0634672Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0634821Z 2025-12-04T09:43:34.0634921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0635265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0635571Z res = mod(**inputs) 2025-12-04T09:43:34.0635952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0636374Z outputs = self.mobilebert( 2025-12-04T09:43:34.0636761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0637154Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0637545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0637945Z layer_outputs = layer_module( 2025-12-04T09:43:34.0638332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0638751Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0639171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0639607Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0640037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0640449Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0640584Z 2025-12-04T09:43:34.0640688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0641032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0641334Z res = mod(**inputs) 2025-12-04T09:43:34.0641705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0642101Z outputs = self.mobilebert( 2025-12-04T09:43:34.0642475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0642872Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0643266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0643665Z layer_outputs = layer_module( 2025-12-04T09:43:34.0644056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0644479Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0644908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0645378Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0645814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0646285Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0646449Z 2025-12-04T09:43:34.0646558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0646919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0647241Z res = mod(**inputs) 2025-12-04T09:43:34.0647625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0648035Z outputs = self.mobilebert( 2025-12-04T09:43:34.0648426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0648839Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0649245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0649660Z layer_outputs = layer_module( 2025-12-04T09:43:34.0650079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0650534Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0650991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0651553Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0652064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0652545Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0652696Z 2025-12-04T09:43:34.0652812Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0653188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0653542Z res = mod(**inputs) 2025-12-04T09:43:34.0653959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0654403Z outputs = self.mobilebert( 2025-12-04T09:43:34.0654822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0655265Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0655696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0656129Z layer_outputs = layer_module( 2025-12-04T09:43:34.0656562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0657019Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0657481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0657942Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0658407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0658863Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0659333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0659757Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0659913Z 2025-12-04T09:43:34.0660041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0660400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0660716Z res = mod(**inputs) 2025-12-04T09:43:34.0661122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0661554Z outputs = self.mobilebert( 2025-12-04T09:43:34.0661957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0662368Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0662776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0663193Z layer_outputs = layer_module( 2025-12-04T09:43:34.0663603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0664058Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0664524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0664965Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0665103Z 2025-12-04T09:43:34.0665204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0665553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0665872Z res = mod(**inputs) 2025-12-04T09:43:34.0666250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0666647Z outputs = self.mobilebert( 2025-12-04T09:43:34.0667041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0667449Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0667849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0668253Z layer_outputs = layer_module( 2025-12-04T09:43:34.0668663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0669120Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0669564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0670010Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0670178Z 2025-12-04T09:43:34.0670280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0670637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0670963Z res = mod(**inputs) 2025-12-04T09:43:34.0671347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0671759Z outputs = self.mobilebert( 2025-12-04T09:43:34.0672154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0672713Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0673128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0673547Z layer_outputs = layer_module( 2025-12-04T09:43:34.0673949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0674503Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0675007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.0675476Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.0675625Z 2025-12-04T09:43:34.0675753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0676112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0676437Z res = mod(**inputs) 2025-12-04T09:43:34.0676819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0677206Z outputs = self.mobilebert( 2025-12-04T09:43:34.0677587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0677991Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0678380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0678807Z layer_outputs = layer_module( 2025-12-04T09:43:34.0679206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0679686Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0680157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.0680599Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.0681039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0681459Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0681601Z 2025-12-04T09:43:34.0683143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0683489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0683799Z res = mod(**inputs) 2025-12-04T09:43:34.0684165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0684551Z outputs = self.mobilebert( 2025-12-04T09:43:34.0684932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0685333Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0685725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0686143Z layer_outputs = layer_module( 2025-12-04T09:43:34.0686536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0687022Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0687512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0687971Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0688426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.0688850Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0688987Z 2025-12-04T09:43:34.0689088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0689463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0689791Z res = mod(**inputs) 2025-12-04T09:43:34.0690171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0690628Z outputs = self.mobilebert( 2025-12-04T09:43:34.0691062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0691630Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0692077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0692539Z layer_outputs = layer_module( 2025-12-04T09:43:34.0692989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0693551Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0694049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0694514Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0695009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.0695488Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0695989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0696463Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0696623Z 2025-12-04T09:43:34.0696743Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0697133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0697497Z res = mod(**inputs) 2025-12-04T09:43:34.0697928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0698391Z outputs = self.mobilebert( 2025-12-04T09:43:34.0698826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0699272Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0699711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0700152Z layer_outputs = layer_module( 2025-12-04T09:43:34.0700581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0701129Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0701679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0702156Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0702644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0703102Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0703239Z 2025-12-04T09:43:34.0703348Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0703693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0704013Z res = mod(**inputs) 2025-12-04T09:43:34.0704400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0704793Z outputs = self.mobilebert( 2025-12-04T09:43:34.0705185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0705588Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0706001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0706414Z layer_outputs = layer_module( 2025-12-04T09:43:34.0706809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0707221Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0707634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0708027Z self_outputs = self.self( 2025-12-04T09:43:34.0708415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.0708815Z self.value(value_tensor) 2025-12-04T09:43:34.0708925Z 2025-12-04T09:43:34.0709031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0709398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0709708Z res = mod(**inputs) 2025-12-04T09:43:34.0710088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0710484Z outputs = self.mobilebert( 2025-12-04T09:43:34.0710868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0711281Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0711675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0712063Z layer_outputs = layer_module( 2025-12-04T09:43:34.0712454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0712943Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0713421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.0713855Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.0714287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0714695Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0714829Z 2025-12-04T09:43:34.0714927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0715277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0715589Z res = mod(**inputs) 2025-12-04T09:43:34.0715964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0716361Z outputs = self.mobilebert( 2025-12-04T09:43:34.0716749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0717147Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0717531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0717927Z layer_outputs = layer_module( 2025-12-04T09:43:34.0718316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0718835Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0719317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0719788Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0720228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.0720644Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.0721050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0721468Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0721607Z 2025-12-04T09:43:34.0721715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0722060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0722365Z res = mod(**inputs) 2025-12-04T09:43:34.0722739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0723192Z outputs = self.mobilebert( 2025-12-04T09:43:34.0723570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0723971Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0724364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0724765Z layer_outputs = layer_module( 2025-12-04T09:43:34.0725152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0725569Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0725982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0726383Z self_outputs = self.self( 2025-12-04T09:43:34.0726767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.0727170Z self.query(query_tensor) 2025-12-04T09:43:34.0727281Z 2025-12-04T09:43:34.0727387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0727727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0728040Z res = mod(**inputs) 2025-12-04T09:43:34.0728415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0728818Z outputs = self.mobilebert( 2025-12-04T09:43:34.0729208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0729623Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0730025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0730427Z layer_outputs = layer_module( 2025-12-04T09:43:34.0730830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0731331Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0731771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0732194Z self_outputs = self.self( 2025-12-04T09:43:34.0732644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.0733079Z self.key(key_tensor) 2025-12-04T09:43:34.0733193Z 2025-12-04T09:43:34.0733298Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0733507Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0733770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0734148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0734469Z res = mod(**inputs) 2025-12-04T09:43:34.0734855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0735261Z outputs = self.mobilebert( 2025-12-04T09:43:34.0735658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0736057Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0736461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0736870Z layer_outputs = layer_module( 2025-12-04T09:43:34.0737258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0737710Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0738118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0738565Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0739000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.0739412Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0739546Z 2025-12-04T09:43:34.0739652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0739994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0740295Z res = mod(**inputs) 2025-12-04T09:43:34.0740668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0741065Z outputs = self.mobilebert( 2025-12-04T09:43:34.0741441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0741838Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0742228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0742628Z layer_outputs = layer_module( 2025-12-04T09:43:34.0743014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0743427Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0743831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0744276Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0744725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.0745171Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0745623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0746054Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0746203Z 2025-12-04T09:43:34.0746301Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0746670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0746984Z res = mod(**inputs) 2025-12-04T09:43:34.0747352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0747769Z outputs = self.mobilebert( 2025-12-04T09:43:34.0748171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0748572Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0748964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0749365Z layer_outputs = layer_module( 2025-12-04T09:43:34.0749759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0750195Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0750628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0751090Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0751552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0751959Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0752100Z 2025-12-04T09:43:34.0752198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0752540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0752852Z res = mod(**inputs) 2025-12-04T09:43:34.0753215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0753613Z outputs = self.mobilebert( 2025-12-04T09:43:34.0753998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0754395Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0754790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0755186Z layer_outputs = layer_module( 2025-12-04T09:43:34.0755576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0755992Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0756410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0756864Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0757298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0757738Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0757910Z 2025-12-04T09:43:34.0758011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0758364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0758675Z res = mod(**inputs) 2025-12-04T09:43:34.0759059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0759468Z outputs = self.mobilebert( 2025-12-04T09:43:34.0759866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0760267Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0761614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0762047Z layer_outputs = layer_module( 2025-12-04T09:43:34.0762447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0762900Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0763356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0763825Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0764280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0764704Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0764847Z 2025-12-04T09:43:34.0764952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0765312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0765624Z res = mod(**inputs) 2025-12-04T09:43:34.0766009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0766440Z outputs = self.mobilebert( 2025-12-04T09:43:34.0766842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0767248Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0767654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0768095Z layer_outputs = layer_module( 2025-12-04T09:43:34.0768513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0768972Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0769430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0769925Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0770403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0770888Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0771453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0771917Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0772076Z 2025-12-04T09:43:34.0772187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0772575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0772651Z res = mod(**inputs) 2025-12-04T09:43:34.0772956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0773044Z outputs = self.mobilebert( 2025-12-04T09:43:34.0773347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0773433Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0773735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0773808Z layer_outputs = layer_module( 2025-12-04T09:43:34.0774108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0774270Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0774573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0774723Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0775072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0775176Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0775180Z 2025-12-04T09:43:34.0775289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0775505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0775571Z res = mod(**inputs) 2025-12-04T09:43:34.0775859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0775940Z outputs = self.mobilebert( 2025-12-04T09:43:34.0776232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0776335Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0776641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0776714Z layer_outputs = layer_module( 2025-12-04T09:43:34.0777018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0777116Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0777413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0777539Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0777837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0777961Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0777969Z 2025-12-04T09:43:34.0778078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0778287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0778363Z res = mod(**inputs) 2025-12-04T09:43:34.0778661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0778736Z outputs = self.mobilebert( 2025-12-04T09:43:34.0779038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0779113Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0779421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0779493Z layer_outputs = layer_module( 2025-12-04T09:43:34.0779791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0779896Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0780191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0780329Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0780625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0780711Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0780715Z 2025-12-04T09:43:34.0780847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0781055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0781122Z res = mod(**inputs) 2025-12-04T09:43:34.0781454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0781546Z outputs = self.mobilebert( 2025-12-04T09:43:34.0781851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0781924Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0782219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0782301Z layer_outputs = layer_module( 2025-12-04T09:43:34.0782601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0782705Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0783003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0783151Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0783452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0783578Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0783878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0783975Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0783979Z 2025-12-04T09:43:34.0784087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0784298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0784364Z res = mod(**inputs) 2025-12-04T09:43:34.0784658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0784740Z outputs = self.mobilebert( 2025-12-04T09:43:34.0785031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0785117Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0785408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0785481Z layer_outputs = layer_module( 2025-12-04T09:43:34.0785781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0785876Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0786175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0786292Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0786589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0786691Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0786695Z 2025-12-04T09:43:34.0786792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0786984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0787054Z res = mod(**inputs) 2025-12-04T09:43:34.0787353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0787433Z outputs = self.mobilebert( 2025-12-04T09:43:34.0787713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0787800Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0788114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0788187Z layer_outputs = layer_module( 2025-12-04T09:43:34.0788476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0788567Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0788842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0788958Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0789236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0789347Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0789378Z 2025-12-04T09:43:34.0789482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0789681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0789752Z res = mod(**inputs) 2025-12-04T09:43:34.0790032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0790102Z outputs = self.mobilebert( 2025-12-04T09:43:34.0790384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0790456Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0790738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0790809Z layer_outputs = layer_module( 2025-12-04T09:43:34.0791087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0791188Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0791463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0791589Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0791870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0791950Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0791955Z 2025-12-04T09:43:34.0792062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0792255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0792320Z res = mod(**inputs) 2025-12-04T09:43:34.0792604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0792673Z outputs = self.mobilebert( 2025-12-04T09:43:34.0792962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0793031Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0793311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0793386Z layer_outputs = layer_module( 2025-12-04T09:43:34.0793685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0793784Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0794064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0794226Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0794512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0794630Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0794908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0795007Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0795010Z 2025-12-04T09:43:34.0795111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0795311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0795375Z res = mod(**inputs) 2025-12-04T09:43:34.0795685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0795764Z outputs = self.mobilebert( 2025-12-04T09:43:34.0796043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0796120Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0796398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0796469Z layer_outputs = layer_module( 2025-12-04T09:43:34.0796752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0796872Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0797150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0797243Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0797247Z 2025-12-04T09:43:34.0797348Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0797546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0797610Z res = mod(**inputs) 2025-12-04T09:43:34.0797888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0797965Z outputs = self.mobilebert( 2025-12-04T09:43:34.0798245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0798322Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0798600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0798670Z layer_outputs = layer_module( 2025-12-04T09:43:34.0798955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0799071Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0799353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0799465Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0799468Z 2025-12-04T09:43:34.0799566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0799782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0799847Z res = mod(**inputs) 2025-12-04T09:43:34.0800124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0800226Z outputs = self.mobilebert( 2025-12-04T09:43:34.0800524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0800604Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0800881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0800950Z layer_outputs = layer_module( 2025-12-04T09:43:34.0801230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0801388Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0801677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.0801790Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.0801794Z 2025-12-04T09:43:34.0801894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0802098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0802159Z res = mod(**inputs) 2025-12-04T09:43:34.0802446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0802519Z outputs = self.mobilebert( 2025-12-04T09:43:34.0802783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0802858Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0803132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0803204Z layer_outputs = layer_module( 2025-12-04T09:43:34.0803487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0803643Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0803925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.0804044Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.0804321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0804421Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0804424Z 2025-12-04T09:43:34.0804523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0804722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0804786Z res = mod(**inputs) 2025-12-04T09:43:34.0805066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0805142Z outputs = self.mobilebert( 2025-12-04T09:43:34.0805416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0805484Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0805768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0805836Z layer_outputs = layer_module( 2025-12-04T09:43:34.0806158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0806312Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0806624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0806756Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0807033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.0807122Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0807125Z 2025-12-04T09:43:34.0807227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0807434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0807506Z res = mod(**inputs) 2025-12-04T09:43:34.0807797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0807891Z outputs = self.mobilebert( 2025-12-04T09:43:34.0808192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0808266Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0808566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0808637Z layer_outputs = layer_module( 2025-12-04T09:43:34.0808929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0809101Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0809391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0809522Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0809818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.0809945Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0810242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0810335Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0810339Z 2025-12-04T09:43:34.0810452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0810658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0810724Z res = mod(**inputs) 2025-12-04T09:43:34.0811022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0811099Z outputs = self.mobilebert( 2025-12-04T09:43:34.0811467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0811562Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0811862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0811945Z layer_outputs = layer_module( 2025-12-04T09:43:34.0812253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0812428Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0812781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0812901Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0813240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0813344Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0813348Z 2025-12-04T09:43:34.0813458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0813675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0813742Z res = mod(**inputs) 2025-12-04T09:43:34.0814036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0814121Z outputs = self.mobilebert( 2025-12-04T09:43:34.0814420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0814504Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0814817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0814895Z layer_outputs = layer_module( 2025-12-04T09:43:34.0815195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0815284Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0815589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0815662Z self_outputs = self.self( 2025-12-04T09:43:34.0816010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.0816095Z self.value(value_tensor) 2025-12-04T09:43:34.0816099Z 2025-12-04T09:43:34.0816205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0816414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0816488Z res = mod(**inputs) 2025-12-04T09:43:34.0816786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0816866Z outputs = self.mobilebert( 2025-12-04T09:43:34.0817161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0817235Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0817538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0817613Z layer_outputs = layer_module( 2025-12-04T09:43:34.0817917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0818089Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0818385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.0818507Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.0818801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0818885Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0818897Z 2025-12-04T09:43:34.0819005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0819227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0819301Z res = mod(**inputs) 2025-12-04T09:43:34.0819601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0819695Z outputs = self.mobilebert( 2025-12-04T09:43:34.0820036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0820113Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0820415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0820486Z layer_outputs = layer_module( 2025-12-04T09:43:34.0820783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0820963Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0821262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0821416Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0821717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.0821805Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.0822111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0822207Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0822211Z 2025-12-04T09:43:34.0822317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0822533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0822599Z res = mod(**inputs) 2025-12-04T09:43:34.0822902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0822979Z outputs = self.mobilebert( 2025-12-04T09:43:34.0823273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0823354Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0823648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0823727Z layer_outputs = layer_module( 2025-12-04T09:43:34.0824022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0824108Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0824413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0824481Z self_outputs = self.self( 2025-12-04T09:43:34.0824758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.0824833Z self.query(query_tensor) 2025-12-04T09:43:34.0824838Z 2025-12-04T09:43:34.0824940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0825138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0825198Z res = mod(**inputs) 2025-12-04T09:43:34.0825472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0825544Z outputs = self.mobilebert( 2025-12-04T09:43:34.0825839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0825917Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0826188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0826274Z layer_outputs = layer_module( 2025-12-04T09:43:34.0826565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0826646Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0826912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0826986Z self_outputs = self.self( 2025-12-04T09:43:34.0827253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.0827324Z self.key(key_tensor) 2025-12-04T09:43:34.0827327Z 2025-12-04T09:43:34.0827404Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0827479Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0827608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0827797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0827861Z res = mod(**inputs) 2025-12-04T09:43:34.0828135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0828201Z outputs = self.mobilebert( 2025-12-04T09:43:34.0828483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0828550Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0828826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0828901Z layer_outputs = layer_module( 2025-12-04T09:43:34.0829171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0829260Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0829532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0829654Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0829937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.0830019Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0830022Z 2025-12-04T09:43:34.0830131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0830329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0830390Z res = mod(**inputs) 2025-12-04T09:43:34.0830674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0830746Z outputs = self.mobilebert( 2025-12-04T09:43:34.0831025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0831104Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0831383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0831458Z layer_outputs = layer_module( 2025-12-04T09:43:34.0831739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0831843Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0832130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0832272Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0832576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.0832702Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0832980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0833079Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0833083Z 2025-12-04T09:43:34.0833182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0833377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0833447Z res = mod(**inputs) 2025-12-04T09:43:34.0833723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0833824Z outputs = self.mobilebert( 2025-12-04T09:43:34.0834126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0834199Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0834504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0834576Z layer_outputs = layer_module( 2025-12-04T09:43:34.0834880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0834981Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0835282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0835408Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0835712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0835801Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0835812Z 2025-12-04T09:43:34.0835920Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0836128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0836203Z res = mod(**inputs) 2025-12-04T09:43:34.0836501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0836575Z outputs = self.mobilebert( 2025-12-04T09:43:34.0836879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0836952Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0837260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0837333Z layer_outputs = layer_module( 2025-12-04T09:43:34.0837632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0837740Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0838040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0838156Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0838483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0838603Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0838624Z 2025-12-04T09:43:34.0838740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0838963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0839031Z res = mod(**inputs) 2025-12-04T09:43:34.0839333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0839405Z outputs = self.mobilebert( 2025-12-04T09:43:34.0839701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0839773Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0840064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0840144Z layer_outputs = layer_module( 2025-12-04T09:43:34.0840441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0840564Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0840865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0840994Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0841297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0841382Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0841386Z 2025-12-04T09:43:34.0841492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0841701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0841767Z res = mod(**inputs) 2025-12-04T09:43:34.0842073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0842146Z outputs = self.mobilebert( 2025-12-04T09:43:34.0842439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0842522Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0842817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0842896Z layer_outputs = layer_module( 2025-12-04T09:43:34.0843194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0843288Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0843589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0843723Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0844023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0844156Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0844452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0844552Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0844556Z 2025-12-04T09:43:34.0844660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0844887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0844963Z res = mod(**inputs) 2025-12-04T09:43:34.0845257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0845365Z outputs = self.mobilebert( 2025-12-04T09:43:34.0845682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0845759Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0846057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0846129Z layer_outputs = layer_module( 2025-12-04T09:43:34.0846423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0846530Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0846825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0846973Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0847268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0847354Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0847358Z 2025-12-04T09:43:34.0847473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0847676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0847749Z res = mod(**inputs) 2025-12-04T09:43:34.0848042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0848116Z outputs = self.mobilebert( 2025-12-04T09:43:34.0848416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0848493Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0848786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0848866Z layer_outputs = layer_module( 2025-12-04T09:43:34.0849157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0849258Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0849553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0849668Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0849966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0850081Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0850088Z 2025-12-04T09:43:34.0850204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0850412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0850478Z res = mod(**inputs) 2025-12-04T09:43:34.0850776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0850849Z outputs = self.mobilebert( 2025-12-04T09:43:34.0851149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0851286Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0851624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0851721Z layer_outputs = layer_module( 2025-12-04T09:43:34.0852040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0852196Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0852511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0852645Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0852963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0853049Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0853053Z 2025-12-04T09:43:34.0853159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0853369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0853434Z res = mod(**inputs) 2025-12-04T09:43:34.0853754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0853828Z outputs = self.mobilebert( 2025-12-04T09:43:34.0854120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0854203Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0854496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0854569Z layer_outputs = layer_module( 2025-12-04T09:43:34.0854870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0854965Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0855264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0855397Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0855690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0855825Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0856118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0856219Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0856223Z 2025-12-04T09:43:34.0856331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0856536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0856612Z res = mod(**inputs) 2025-12-04T09:43:34.0856907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0856990Z outputs = self.mobilebert( 2025-12-04T09:43:34.0857287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0857362Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0857663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0857737Z layer_outputs = layer_module( 2025-12-04T09:43:34.0858053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0858159Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0858447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0858583Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0858874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0858956Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0858959Z 2025-12-04T09:43:34.0859067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0859258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0859326Z res = mod(**inputs) 2025-12-04T09:43:34.0859599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0859666Z outputs = self.mobilebert( 2025-12-04T09:43:34.0859945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0860034Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0860313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0860388Z layer_outputs = layer_module( 2025-12-04T09:43:34.0860662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0860756Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0861029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0861137Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0861417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0861528Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0861533Z 2025-12-04T09:43:34.0861637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0861831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0861894Z res = mod(**inputs) 2025-12-04T09:43:34.0862175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0862245Z outputs = self.mobilebert( 2025-12-04T09:43:34.0862518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0862598Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0862872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0862948Z layer_outputs = layer_module( 2025-12-04T09:43:34.0863226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0863317Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0863604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0863725Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0864006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0864086Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0864089Z 2025-12-04T09:43:34.0864207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0864409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0864491Z res = mod(**inputs) 2025-12-04T09:43:34.0864776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0864860Z outputs = self.mobilebert( 2025-12-04T09:43:34.0865136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0865213Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0865491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0865561Z layer_outputs = layer_module( 2025-12-04T09:43:34.0865849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0865938Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0866220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0866361Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0866638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0866764Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0867039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0867137Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0867141Z 2025-12-04T09:43:34.0867240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0867435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0867506Z res = mod(**inputs) 2025-12-04T09:43:34.0867785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0867857Z outputs = self.mobilebert( 2025-12-04T09:43:34.0868142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0868212Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0868493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0868563Z layer_outputs = layer_module( 2025-12-04T09:43:34.0868850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0868976Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0869253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0869344Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0869347Z 2025-12-04T09:43:34.0869446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0869638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0869706Z res = mod(**inputs) 2025-12-04T09:43:34.0869985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0870053Z outputs = self.mobilebert( 2025-12-04T09:43:34.0870365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0870439Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0870724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0870813Z layer_outputs = layer_module( 2025-12-04T09:43:34.0871105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0871233Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0871508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0871622Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0871626Z 2025-12-04T09:43:34.0871726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0871923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0871993Z res = mod(**inputs) 2025-12-04T09:43:34.0872420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0872560Z outputs = self.mobilebert( 2025-12-04T09:43:34.0872842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0872913Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0873200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0873268Z layer_outputs = layer_module( 2025-12-04T09:43:34.0873544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0873711Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0873990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.0874093Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.0874098Z 2025-12-04T09:43:34.0874199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0874395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0874465Z res = mod(**inputs) 2025-12-04T09:43:34.0874740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0874814Z outputs = self.mobilebert( 2025-12-04T09:43:34.0875092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0875162Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0875450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0875520Z layer_outputs = layer_module( 2025-12-04T09:43:34.0875800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0875963Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0876239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.0876362Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.0876638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0876754Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0876758Z 2025-12-04T09:43:34.0876869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0877061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0877162Z res = mod(**inputs) 2025-12-04T09:43:34.0877463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0877533Z outputs = self.mobilebert( 2025-12-04T09:43:34.0877820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0877890Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0878171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0878248Z layer_outputs = layer_module( 2025-12-04T09:43:34.0878528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0878690Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0878987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0879109Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0879397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.0879480Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0879483Z 2025-12-04T09:43:34.0879591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0879784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0879848Z res = mod(**inputs) 2025-12-04T09:43:34.0880137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0880209Z outputs = self.mobilebert( 2025-12-04T09:43:34.0880493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0880562Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0880837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0880913Z layer_outputs = layer_module( 2025-12-04T09:43:34.0881187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0881337Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0881623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0881740Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0882024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.0882146Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0882422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0882523Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0882527Z 2025-12-04T09:43:34.0882632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0882850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0882935Z res = mod(**inputs) 2025-12-04T09:43:34.0883232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0883359Z outputs = self.mobilebert( 2025-12-04T09:43:34.0883652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0883748Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0884042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0884116Z layer_outputs = layer_module( 2025-12-04T09:43:34.0884412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0884580Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0884884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0885005Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0885329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0885422Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0885426Z 2025-12-04T09:43:34.0885530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0885733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0885805Z res = mod(**inputs) 2025-12-04T09:43:34.0886095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0886173Z outputs = self.mobilebert( 2025-12-04T09:43:34.0886466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0886539Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0886840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0886915Z layer_outputs = layer_module( 2025-12-04T09:43:34.0887207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0887302Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0887593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0887672Z self_outputs = self.self( 2025-12-04T09:43:34.0887964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.0888037Z self.value(value_tensor) 2025-12-04T09:43:34.0888041Z 2025-12-04T09:43:34.0888154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0888360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0888434Z res = mod(**inputs) 2025-12-04T09:43:34.0888726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0888797Z outputs = self.mobilebert( 2025-12-04T09:43:34.0889094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0889167Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0889454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0889555Z layer_outputs = layer_module( 2025-12-04T09:43:34.0889848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0890043Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0890352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.0890467Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.0890766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0890850Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0890853Z 2025-12-04T09:43:34.0890967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0891178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0891308Z res = mod(**inputs) 2025-12-04T09:43:34.0891625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0891736Z outputs = self.mobilebert( 2025-12-04T09:43:34.0892045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0892128Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0892442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0892523Z layer_outputs = layer_module( 2025-12-04T09:43:34.0892825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0892997Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0893308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0893425Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0893736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.0893826Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.0894123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0894229Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0894233Z 2025-12-04T09:43:34.0894340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0894568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0894637Z res = mod(**inputs) 2025-12-04T09:43:34.0894934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0895017Z outputs = self.mobilebert( 2025-12-04T09:43:34.0895317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0895391Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0895699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0895771Z layer_outputs = layer_module( 2025-12-04T09:43:34.0896076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0896161Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0896480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0896562Z self_outputs = self.self( 2025-12-04T09:43:34.0896882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.0896967Z self.query(query_tensor) 2025-12-04T09:43:34.0896989Z 2025-12-04T09:43:34.0897098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0897309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0897385Z res = mod(**inputs) 2025-12-04T09:43:34.0897684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0897756Z outputs = self.mobilebert( 2025-12-04T09:43:34.0898063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0898136Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0898438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0898535Z layer_outputs = layer_module( 2025-12-04T09:43:34.0898831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0898925Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0899218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0899299Z self_outputs = self.self( 2025-12-04T09:43:34.0899592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.0899663Z self.key(key_tensor) 2025-12-04T09:43:34.0899667Z 2025-12-04T09:43:34.0899758Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0899841Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0899953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0900179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0900244Z res = mod(**inputs) 2025-12-04T09:43:34.0900525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0900594Z outputs = self.mobilebert( 2025-12-04T09:43:34.0900869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0900946Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0901224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0901293Z layer_outputs = layer_module( 2025-12-04T09:43:34.0901578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0901663Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0901945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0902068Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0902343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.0902432Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0902436Z 2025-12-04T09:43:34.0902536Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0902756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0902819Z res = mod(**inputs) 2025-12-04T09:43:34.0903105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0903200Z outputs = self.mobilebert( 2025-12-04T09:43:34.0903496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0903566Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0903853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0903921Z layer_outputs = layer_module( 2025-12-04T09:43:34.0904205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0904288Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0904562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0904712Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0904996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.0905129Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0905410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0905501Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0905505Z 2025-12-04T09:43:34.0905614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0905811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0905882Z res = mod(**inputs) 2025-12-04T09:43:34.0906159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0906232Z outputs = self.mobilebert( 2025-12-04T09:43:34.0906518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0906588Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0906869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0906946Z layer_outputs = layer_module( 2025-12-04T09:43:34.0907222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0907325Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0907606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0907718Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0908006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0908088Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0908092Z 2025-12-04T09:43:34.0908197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0908392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0908454Z res = mod(**inputs) 2025-12-04T09:43:34.0908737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0908825Z outputs = self.mobilebert( 2025-12-04T09:43:34.0909102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0909181Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0909478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0909571Z layer_outputs = layer_module( 2025-12-04T09:43:34.0909849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0909941Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0910222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0910328Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0910608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0910718Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0910723Z 2025-12-04T09:43:34.0910841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0911046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0911109Z res = mod(**inputs) 2025-12-04T09:43:34.0911383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0911459Z outputs = self.mobilebert( 2025-12-04T09:43:34.0911739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0911814Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0912093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0912162Z layer_outputs = layer_module( 2025-12-04T09:43:34.0912446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0912540Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0912825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0912952Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0913231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0913322Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0913325Z 2025-12-04T09:43:34.0913426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0913628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0913689Z res = mod(**inputs) 2025-12-04T09:43:34.0913962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0914039Z outputs = self.mobilebert( 2025-12-04T09:43:34.0914319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0914387Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0914667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0914735Z layer_outputs = layer_module( 2025-12-04T09:43:34.0915035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0915131Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0915407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0915555Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0915856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0915985Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0916260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0916350Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0916354Z 2025-12-04T09:43:34.0916461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0916656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0916719Z res = mod(**inputs) 2025-12-04T09:43:34.0917007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0917096Z outputs = self.mobilebert( 2025-12-04T09:43:34.0917386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0917457Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0917737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0917822Z layer_outputs = layer_module( 2025-12-04T09:43:34.0918102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0918200Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0918480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0918592Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0918876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0918957Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0918961Z 2025-12-04T09:43:34.0919067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0919262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0919324Z res = mod(**inputs) 2025-12-04T09:43:34.0919609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0919678Z outputs = self.mobilebert( 2025-12-04T09:43:34.0919959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0920038Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0920319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0920395Z layer_outputs = layer_module( 2025-12-04T09:43:34.0920677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0920767Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0921049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0921156Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0921461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0921573Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0921593Z 2025-12-04T09:43:34.0921695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0921910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0921976Z res = mod(**inputs) 2025-12-04T09:43:34.0922260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0922338Z outputs = self.mobilebert( 2025-12-04T09:43:34.0922618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0922694Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0922985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0923053Z layer_outputs = layer_module( 2025-12-04T09:43:34.0923341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0923449Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0923739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0923859Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0924137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0924224Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0924228Z 2025-12-04T09:43:34.0924328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0924522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0924591Z res = mod(**inputs) 2025-12-04T09:43:34.0924874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0924948Z outputs = self.mobilebert( 2025-12-04T09:43:34.0925226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0925295Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0925578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0925646Z layer_outputs = layer_module( 2025-12-04T09:43:34.0925932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0926021Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0926299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0926434Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0926728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0926853Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0927155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0927248Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0927251Z 2025-12-04T09:43:34.0927364Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0927587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0927654Z res = mod(**inputs) 2025-12-04T09:43:34.0927957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0928050Z outputs = self.mobilebert( 2025-12-04T09:43:34.0928365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0928441Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0928729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0928811Z layer_outputs = layer_module( 2025-12-04T09:43:34.0929104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0929204Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0929495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0929628Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0929932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0930019Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0930022Z 2025-12-04T09:43:34.0930128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0930339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0930405Z res = mod(**inputs) 2025-12-04T09:43:34.0930704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0930778Z outputs = self.mobilebert( 2025-12-04T09:43:34.0931068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0931154Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0931664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0931755Z layer_outputs = layer_module( 2025-12-04T09:43:34.0932057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0932156Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0932470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0932589Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0932880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0933007Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0933013Z 2025-12-04T09:43:34.0933120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0933337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0933403Z res = mod(**inputs) 2025-12-04T09:43:34.0933697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0933781Z outputs = self.mobilebert( 2025-12-04T09:43:34.0934075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0934178Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0934473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0934546Z layer_outputs = layer_module( 2025-12-04T09:43:34.0934867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0934984Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0935276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0935414Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0935707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0935797Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0935800Z 2025-12-04T09:43:34.0935906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0936109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0936183Z res = mod(**inputs) 2025-12-04T09:43:34.0936497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0936578Z outputs = self.mobilebert( 2025-12-04T09:43:34.0936873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0936946Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0937248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0937320Z layer_outputs = layer_module( 2025-12-04T09:43:34.0937614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0937716Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0938011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0938146Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0938441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0938567Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0938870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0938965Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0938969Z 2025-12-04T09:43:34.0939083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0939291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0939358Z res = mod(**inputs) 2025-12-04T09:43:34.0939662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0939734Z outputs = self.mobilebert( 2025-12-04T09:43:34.0940039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0940111Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0940405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0940485Z layer_outputs = layer_module( 2025-12-04T09:43:34.0940797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0940933Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0941220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0941330Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0941334Z 2025-12-04T09:43:34.0941456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0941650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0941712Z res = mod(**inputs) 2025-12-04T09:43:34.0942000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0942069Z outputs = self.mobilebert( 2025-12-04T09:43:34.0942356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0942425Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0942703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0942797Z layer_outputs = layer_module( 2025-12-04T09:43:34.0943084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.0943201Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.0943494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0943602Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0943605Z 2025-12-04T09:43:34.0943711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0943912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0943974Z res = mod(**inputs) 2025-12-04T09:43:34.0944265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0944337Z outputs = self.mobilebert( 2025-12-04T09:43:34.0944629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0944700Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0944984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0945060Z layer_outputs = layer_module( 2025-12-04T09:43:34.0945346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0945512Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0945798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.0945892Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.0945896Z 2025-12-04T09:43:34.0946004Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0946203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0946264Z res = mod(**inputs) 2025-12-04T09:43:34.0946559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0946627Z outputs = self.mobilebert( 2025-12-04T09:43:34.0946917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0947005Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0947278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0947375Z layer_outputs = layer_module( 2025-12-04T09:43:34.0947651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0947829Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0948105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.0948226Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.0948508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0948600Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0948603Z 2025-12-04T09:43:34.0948711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0948904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0948996Z res = mod(**inputs) 2025-12-04T09:43:34.0949285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0949355Z outputs = self.mobilebert( 2025-12-04T09:43:34.0949630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0949709Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0949985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0950061Z layer_outputs = layer_module( 2025-12-04T09:43:34.0950341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0950496Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0950783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0950905Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0951188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.0951269Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0951273Z 2025-12-04T09:43:34.0951377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0951577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0951638Z res = mod(**inputs) 2025-12-04T09:43:34.0951912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0951993Z outputs = self.mobilebert( 2025-12-04T09:43:34.0952269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0952347Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0952624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0952692Z layer_outputs = layer_module( 2025-12-04T09:43:34.0952978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.0953129Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.0953428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.0953552Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.0953863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.0954007Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0954286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0954381Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0954385Z 2025-12-04T09:43:34.0954483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0954684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0954752Z res = mod(**inputs) 2025-12-04T09:43:34.0955020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0955088Z outputs = self.mobilebert( 2025-12-04T09:43:34.0955381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0955453Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0955729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0955795Z layer_outputs = layer_module( 2025-12-04T09:43:34.0956063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0956220Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0956489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0956601Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0956870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0956948Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0956952Z 2025-12-04T09:43:34.0957058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0957246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0957305Z res = mod(**inputs) 2025-12-04T09:43:34.0957579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0957644Z outputs = self.mobilebert( 2025-12-04T09:43:34.0957917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0957984Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0958261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0958335Z layer_outputs = layer_module( 2025-12-04T09:43:34.0958605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0958691Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0958956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0959022Z self_outputs = self.self( 2025-12-04T09:43:34.0959315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.0959385Z self.value(value_tensor) 2025-12-04T09:43:34.0959388Z 2025-12-04T09:43:34.0959486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0959698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0959759Z res = mod(**inputs) 2025-12-04T09:43:34.0960051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0960117Z outputs = self.mobilebert( 2025-12-04T09:43:34.0960387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0960463Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0960735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0960810Z layer_outputs = layer_module( 2025-12-04T09:43:34.0961081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0961253Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0961534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.0961638Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.0961910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.0961996Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.0961999Z 2025-12-04T09:43:34.0962097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0962293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0962354Z res = mod(**inputs) 2025-12-04T09:43:34.0962624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0962702Z outputs = self.mobilebert( 2025-12-04T09:43:34.0962973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0963048Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0963314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0963380Z layer_outputs = layer_module( 2025-12-04T09:43:34.0963656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.0963807Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.0964085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.0964190Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.0964461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.0964549Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.0964821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0964908Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0964919Z 2025-12-04T09:43:34.0965017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0965234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0965303Z res = mod(**inputs) 2025-12-04T09:43:34.0965575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0965658Z outputs = self.mobilebert( 2025-12-04T09:43:34.0965950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0966022Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0966306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0966374Z layer_outputs = layer_module( 2025-12-04T09:43:34.0966644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0966734Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0967004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0967071Z self_outputs = self.self( 2025-12-04T09:43:34.0967349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.0967433Z self.query(query_tensor) 2025-12-04T09:43:34.0967438Z 2025-12-04T09:43:34.0967544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0967739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0967799Z res = mod(**inputs) 2025-12-04T09:43:34.0968085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0968154Z outputs = self.mobilebert( 2025-12-04T09:43:34.0968447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0968515Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0968795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0968873Z layer_outputs = layer_module( 2025-12-04T09:43:34.0969153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0969233Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0969520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.0969587Z self_outputs = self.self( 2025-12-04T09:43:34.0969877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.0969942Z self.key(key_tensor) 2025-12-04T09:43:34.0969946Z 2025-12-04T09:43:34.0970024Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0970111Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.0970213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0970406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0970476Z res = mod(**inputs) 2025-12-04T09:43:34.0970754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0970827Z outputs = self.mobilebert( 2025-12-04T09:43:34.0971110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0971178Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0971544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0971622Z layer_outputs = layer_module( 2025-12-04T09:43:34.0971905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0972003Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0972447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0972595Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0972886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.0972970Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0972982Z 2025-12-04T09:43:34.0973089Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0973296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0973371Z res = mod(**inputs) 2025-12-04T09:43:34.0973670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0973762Z outputs = self.mobilebert( 2025-12-04T09:43:34.0974044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0974112Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0974390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0974457Z layer_outputs = layer_module( 2025-12-04T09:43:34.0974728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.0974822Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.0975117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.0975249Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.0975554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.0975684Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0975983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0976079Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0976083Z 2025-12-04T09:43:34.0976189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0976405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0976473Z res = mod(**inputs) 2025-12-04T09:43:34.0976772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0976847Z outputs = self.mobilebert( 2025-12-04T09:43:34.0977139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0977220Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0977512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0977590Z layer_outputs = layer_module( 2025-12-04T09:43:34.0977884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0978009Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0978310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0978425Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0978747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0978859Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0978863Z 2025-12-04T09:43:34.0978970Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0979186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0979250Z res = mod(**inputs) 2025-12-04T09:43:34.0979541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0979620Z outputs = self.mobilebert( 2025-12-04T09:43:34.0979916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0979996Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0980308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0980383Z layer_outputs = layer_module( 2025-12-04T09:43:34.0980683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0980781Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0981074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0981198Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0981495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0981618Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0981623Z 2025-12-04T09:43:34.0981733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0981939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0982015Z res = mod(**inputs) 2025-12-04T09:43:34.0982309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0982388Z outputs = self.mobilebert( 2025-12-04T09:43:34.0982688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0982755Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0983029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0983096Z layer_outputs = layer_module( 2025-12-04T09:43:34.0983362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0983473Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0983744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0983870Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0984138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0984217Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0984220Z 2025-12-04T09:43:34.0984328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0984538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0984610Z res = mod(**inputs) 2025-12-04T09:43:34.0984884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0984971Z outputs = self.mobilebert( 2025-12-04T09:43:34.0985267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0985336Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0985612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0985678Z layer_outputs = layer_module( 2025-12-04T09:43:34.0985949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0986045Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0986323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0986470Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0986749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0986865Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0987140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0987227Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0987231Z 2025-12-04T09:43:34.0987328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0987527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0987588Z res = mod(**inputs) 2025-12-04T09:43:34.0987865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0987936Z outputs = self.mobilebert( 2025-12-04T09:43:34.0988206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0988282Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0988547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0988614Z layer_outputs = layer_module( 2025-12-04T09:43:34.0988886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0988975Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0989248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0989355Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0989623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.0989710Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.0989713Z 2025-12-04T09:43:34.0989808Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0990003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0990064Z res = mod(**inputs) 2025-12-04T09:43:34.0990329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0990417Z outputs = self.mobilebert( 2025-12-04T09:43:34.0990686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0990778Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0991053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0991136Z layer_outputs = layer_module( 2025-12-04T09:43:34.0991412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0991500Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0991766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0991880Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.0992148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.0992263Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.0992285Z 2025-12-04T09:43:34.0992385Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0992574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0992641Z res = mod(**inputs) 2025-12-04T09:43:34.0992912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0992987Z outputs = self.mobilebert( 2025-12-04T09:43:34.0993258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0993326Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0993607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0993674Z layer_outputs = layer_module( 2025-12-04T09:43:34.0993949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0994047Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0994319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0994446Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0994716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.0994797Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.0994800Z 2025-12-04T09:43:34.0994910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0995100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0995169Z res = mod(**inputs) 2025-12-04T09:43:34.0995443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0995521Z outputs = self.mobilebert( 2025-12-04T09:43:34.0995792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0995861Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0996124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0996197Z layer_outputs = layer_module( 2025-12-04T09:43:34.0996481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0996575Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0996840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.0996978Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.0997259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.0997374Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.0997645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.0997730Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.0997733Z 2025-12-04T09:43:34.0997827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.0998019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.0998077Z res = mod(**inputs) 2025-12-04T09:43:34.0998344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.0998431Z outputs = self.mobilebert( 2025-12-04T09:43:34.0998699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.0998772Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.0999038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.0999103Z layer_outputs = layer_module( 2025-12-04T09:43:34.0999376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.0999461Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.0999732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.0999838Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1000105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1000188Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1000192Z 2025-12-04T09:43:34.1000284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1000474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1000531Z res = mod(**inputs) 2025-12-04T09:43:34.1000795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1000867Z outputs = self.mobilebert( 2025-12-04T09:43:34.1001132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1001201Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1001475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1001540Z layer_outputs = layer_module( 2025-12-04T09:43:34.1001812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1001897Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1002176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1002306Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1002570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1002681Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1002702Z 2025-12-04T09:43:34.1002799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1002998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1003068Z res = mod(**inputs) 2025-12-04T09:43:34.1003336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1003404Z outputs = self.mobilebert( 2025-12-04T09:43:34.1003675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1003742Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1004020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1004086Z layer_outputs = layer_module( 2025-12-04T09:43:34.1004362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1004459Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1004721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1004839Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1005103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1005180Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1005183Z 2025-12-04T09:43:34.1005285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1005469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1005538Z res = mod(**inputs) 2025-12-04T09:43:34.1005805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1005873Z outputs = self.mobilebert( 2025-12-04T09:43:34.1006147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1006214Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1006482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1006556Z layer_outputs = layer_module( 2025-12-04T09:43:34.1006823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1006917Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1007183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1007303Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1007576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1007689Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1007961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1008047Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1008050Z 2025-12-04T09:43:34.1008161Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1008359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1008417Z res = mod(**inputs) 2025-12-04T09:43:34.1008705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1008781Z outputs = self.mobilebert( 2025-12-04T09:43:34.1009076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1009155Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1009422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1009490Z layer_outputs = layer_module( 2025-12-04T09:43:34.1009767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1009880Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1010156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1010541Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1010544Z 2025-12-04T09:43:34.1010647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1010846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1010907Z res = mod(**inputs) 2025-12-04T09:43:34.1011285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1011381Z outputs = self.mobilebert( 2025-12-04T09:43:34.1011689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1011774Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1012076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1012154Z layer_outputs = layer_module( 2025-12-04T09:43:34.1012472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1012588Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1012874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1012984Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1012988Z 2025-12-04T09:43:34.1013090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1013294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1013359Z res = mod(**inputs) 2025-12-04T09:43:34.1013636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1013718Z outputs = self.mobilebert( 2025-12-04T09:43:34.1014014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1014098Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1014400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1014468Z layer_outputs = layer_module( 2025-12-04T09:43:34.1014757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1014935Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1015221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1015333Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1015337Z 2025-12-04T09:43:34.1015438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1015655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1015717Z res = mod(**inputs) 2025-12-04T09:43:34.1016009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1016077Z outputs = self.mobilebert( 2025-12-04T09:43:34.1016359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1016438Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1016724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1016793Z layer_outputs = layer_module( 2025-12-04T09:43:34.1017102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1017258Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1017542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1017661Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1017942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1018045Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1018048Z 2025-12-04T09:43:34.1018147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1018346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1018412Z res = mod(**inputs) 2025-12-04T09:43:34.1018687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1018765Z outputs = self.mobilebert( 2025-12-04T09:43:34.1019040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1019110Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1019395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1019464Z layer_outputs = layer_module( 2025-12-04T09:43:34.1019748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1019901Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1020179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1020310Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1020585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1020673Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1020677Z 2025-12-04T09:43:34.1020776Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1020970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1021056Z res = mod(**inputs) 2025-12-04T09:43:34.1021333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1021428Z outputs = self.mobilebert( 2025-12-04T09:43:34.1021709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1021797Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1022091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1022164Z layer_outputs = layer_module( 2025-12-04T09:43:34.1022453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1022628Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1022903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1023026Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1023319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1023439Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1023734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1023822Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1023826Z 2025-12-04T09:43:34.1023933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1024125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1024189Z res = mod(**inputs) 2025-12-04T09:43:34.1024472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1024541Z outputs = self.mobilebert( 2025-12-04T09:43:34.1024819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1024898Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1025173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1025248Z layer_outputs = layer_module( 2025-12-04T09:43:34.1025537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1025704Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1025984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1026089Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1026369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1026449Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1026453Z 2025-12-04T09:43:34.1026549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1026755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1026816Z res = mod(**inputs) 2025-12-04T09:43:34.1027097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1027167Z outputs = self.mobilebert( 2025-12-04T09:43:34.1027460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1027540Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1027832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1027902Z layer_outputs = layer_module( 2025-12-04T09:43:34.1028202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1028288Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1028574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1028643Z self_outputs = self.self( 2025-12-04T09:43:34.1028923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1029002Z self.value(value_tensor) 2025-12-04T09:43:34.1029005Z 2025-12-04T09:43:34.1029104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1029309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1029391Z res = mod(**inputs) 2025-12-04T09:43:34.1029680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1029757Z outputs = self.mobilebert( 2025-12-04T09:43:34.1030042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1030113Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1030403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1030474Z layer_outputs = layer_module( 2025-12-04T09:43:34.1030763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1030926Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1031210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1031327Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1031609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1031695Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1031699Z 2025-12-04T09:43:34.1031799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1031996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1032065Z res = mod(**inputs) 2025-12-04T09:43:34.1032348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1032420Z outputs = self.mobilebert( 2025-12-04T09:43:34.1032711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1032780Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1033073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1033142Z layer_outputs = layer_module( 2025-12-04T09:43:34.1033426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1033605Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1033887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1034027Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1034320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1034404Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1034690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1034780Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1034784Z 2025-12-04T09:43:34.1034892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1035086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1035148Z res = mod(**inputs) 2025-12-04T09:43:34.1035429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1035520Z outputs = self.mobilebert( 2025-12-04T09:43:34.1035799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1035877Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1036154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1036228Z layer_outputs = layer_module( 2025-12-04T09:43:34.1036502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1036584Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1036869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1036936Z self_outputs = self.self( 2025-12-04T09:43:34.1037222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1037294Z self.query(query_tensor) 2025-12-04T09:43:34.1037298Z 2025-12-04T09:43:34.1037400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1037600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1037661Z res = mod(**inputs) 2025-12-04T09:43:34.1037942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1038018Z outputs = self.mobilebert( 2025-12-04T09:43:34.1038297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1038371Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1038652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1038725Z layer_outputs = layer_module( 2025-12-04T09:43:34.1039009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1039089Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1039369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1039436Z self_outputs = self.self( 2025-12-04T09:43:34.1039711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1039800Z self.key(key_tensor) 2025-12-04T09:43:34.1039804Z 2025-12-04T09:43:34.1039887Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1039966Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1040097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1040291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1040375Z res = mod(**inputs) 2025-12-04T09:43:34.1040654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1040721Z outputs = self.mobilebert( 2025-12-04T09:43:34.1041007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1041075Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1041359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1041433Z layer_outputs = layer_module( 2025-12-04T09:43:34.1041711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1041818Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1042095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1042214Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1042539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1042616Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1042619Z 2025-12-04T09:43:34.1042719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1042908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1042968Z res = mod(**inputs) 2025-12-04T09:43:34.1043236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1043304Z outputs = self.mobilebert( 2025-12-04T09:43:34.1043568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1043642Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1043905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1043976Z layer_outputs = layer_module( 2025-12-04T09:43:34.1044239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1044315Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1044585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1044700Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1044971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1045085Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1045346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1045439Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1045442Z 2025-12-04T09:43:34.1045537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1045743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1045804Z res = mod(**inputs) 2025-12-04T09:43:34.1046073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1046166Z outputs = self.mobilebert( 2025-12-04T09:43:34.1046449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1046518Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1046790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1046856Z layer_outputs = layer_module( 2025-12-04T09:43:34.1047135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1047223Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1047487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1047598Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1047879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1047966Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1047969Z 2025-12-04T09:43:34.1048062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1048245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1048310Z res = mod(**inputs) 2025-12-04T09:43:34.1048574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1048639Z outputs = self.mobilebert( 2025-12-04T09:43:34.1048911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1048978Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1049253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1049321Z layer_outputs = layer_module( 2025-12-04T09:43:34.1049585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1049678Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1049941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1050049Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1050313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1050415Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1050420Z 2025-12-04T09:43:34.1050524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1050707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1050767Z res = mod(**inputs) 2025-12-04T09:43:34.1051035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1051100Z outputs = self.mobilebert( 2025-12-04T09:43:34.1051468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1051553Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1051915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1052002Z layer_outputs = layer_module( 2025-12-04T09:43:34.1052304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1052438Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1052762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1052903Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1053179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1053258Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1053261Z 2025-12-04T09:43:34.1053367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1053555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1053617Z res = mod(**inputs) 2025-12-04T09:43:34.1053892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1053976Z outputs = self.mobilebert( 2025-12-04T09:43:34.1054255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1054328Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1054589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1054661Z layer_outputs = layer_module( 2025-12-04T09:43:34.1054925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1055011Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1055281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1055397Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1055667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1055779Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1056041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1056134Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1056138Z 2025-12-04T09:43:34.1056233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1056418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1056484Z res = mod(**inputs) 2025-12-04T09:43:34.1056748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1056826Z outputs = self.mobilebert( 2025-12-04T09:43:34.1057092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1057158Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1057430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1057494Z layer_outputs = layer_module( 2025-12-04T09:43:34.1057764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1057872Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1058135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1058262Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1058541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1058620Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1058630Z 2025-12-04T09:43:34.1058727Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1058910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1058976Z res = mod(**inputs) 2025-12-04T09:43:34.1059246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1059312Z outputs = self.mobilebert( 2025-12-04T09:43:34.1059581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1059649Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1059937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1060005Z layer_outputs = layer_module( 2025-12-04T09:43:34.1060268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1060362Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1060638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1060749Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1061011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1061115Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1061121Z 2025-12-04T09:43:34.1061223Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1061408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1061467Z res = mod(**inputs) 2025-12-04T09:43:34.1061736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1061800Z outputs = self.mobilebert( 2025-12-04T09:43:34.1062075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1062139Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1062397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1062468Z layer_outputs = layer_module( 2025-12-04T09:43:34.1062722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1062814Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1063068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1063179Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1063441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1063515Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1063519Z 2025-12-04T09:43:34.1063627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1063816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1063874Z res = mod(**inputs) 2025-12-04T09:43:34.1064158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1064223Z outputs = self.mobilebert( 2025-12-04T09:43:34.1064498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1064570Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1064825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1064895Z layer_outputs = layer_module( 2025-12-04T09:43:34.1065154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1065237Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1065501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1065632Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1065888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1066004Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1066260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1066350Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1066353Z 2025-12-04T09:43:34.1066446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1066624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1066688Z res = mod(**inputs) 2025-12-04T09:43:34.1066945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1067018Z outputs = self.mobilebert( 2025-12-04T09:43:34.1067273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1067337Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1067599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1067662Z layer_outputs = layer_module( 2025-12-04T09:43:34.1067913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1068004Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1068257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1068364Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1068621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1068697Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1068700Z 2025-12-04T09:43:34.1068798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1068973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1069037Z res = mod(**inputs) 2025-12-04T09:43:34.1069310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1069377Z outputs = self.mobilebert( 2025-12-04T09:43:34.1069648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1069730Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1070014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1070080Z layer_outputs = layer_module( 2025-12-04T09:43:34.1070342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1070435Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1070695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1070800Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1071070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1071172Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1071194Z 2025-12-04T09:43:34.1071298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1071489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1071548Z res = mod(**inputs) 2025-12-04T09:43:34.1071823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1071889Z outputs = self.mobilebert( 2025-12-04T09:43:34.1072159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1072225Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1072630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1072710Z layer_outputs = layer_module( 2025-12-04T09:43:34.1072979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1073068Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1073336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1073452Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1073723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1073802Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1073805Z 2025-12-04T09:43:34.1073904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1074095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1074157Z res = mod(**inputs) 2025-12-04T09:43:34.1074426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1074496Z outputs = self.mobilebert( 2025-12-04T09:43:34.1074759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1074836Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1075096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1075161Z layer_outputs = layer_module( 2025-12-04T09:43:34.1075472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1075561Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1075830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1075973Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1076264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1076388Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1076654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1076749Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1076752Z 2025-12-04T09:43:34.1076852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1077036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1077105Z res = mod(**inputs) 2025-12-04T09:43:34.1077370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1077472Z outputs = self.mobilebert( 2025-12-04T09:43:34.1077734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1077801Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1078071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1078135Z layer_outputs = layer_module( 2025-12-04T09:43:34.1078400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1078521Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1078786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1078872Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1078875Z 2025-12-04T09:43:34.1078972Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1079153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1079219Z res = mod(**inputs) 2025-12-04T09:43:34.1079481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1079556Z outputs = self.mobilebert( 2025-12-04T09:43:34.1079828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1079897Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1080169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1080238Z layer_outputs = layer_module( 2025-12-04T09:43:34.1080509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1080632Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1080909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1081019Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1081023Z 2025-12-04T09:43:34.1081118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1081315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1081384Z res = mod(**inputs) 2025-12-04T09:43:34.1081650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1081743Z outputs = self.mobilebert( 2025-12-04T09:43:34.1082025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1082094Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1082364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1082429Z layer_outputs = layer_module( 2025-12-04T09:43:34.1082696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1082857Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1083129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1083244Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1083247Z 2025-12-04T09:43:34.1083346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1083537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1083606Z res = mod(**inputs) 2025-12-04T09:43:34.1083875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1083947Z outputs = self.mobilebert( 2025-12-04T09:43:34.1084214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1084281Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1084556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1084625Z layer_outputs = layer_module( 2025-12-04T09:43:34.1084901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1085051Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1085317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1085438Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1085705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1085792Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1085803Z 2025-12-04T09:43:34.1085899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1086088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1086158Z res = mod(**inputs) 2025-12-04T09:43:34.1086435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1086502Z outputs = self.mobilebert( 2025-12-04T09:43:34.1086783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1086851Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1087136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1087202Z layer_outputs = layer_module( 2025-12-04T09:43:34.1087492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1087654Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1087955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1088096Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1088378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1088460Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1088463Z 2025-12-04T09:43:34.1088574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1088769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1088833Z res = mod(**inputs) 2025-12-04T09:43:34.1089126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1089201Z outputs = self.mobilebert( 2025-12-04T09:43:34.1089519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1089595Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1089890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1089971Z layer_outputs = layer_module( 2025-12-04T09:43:34.1090263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1090428Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1090722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1090847Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1091165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1091358Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1091681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1091788Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1091792Z 2025-12-04T09:43:34.1091904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1092123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1092193Z res = mod(**inputs) 2025-12-04T09:43:34.1092518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1092594Z outputs = self.mobilebert( 2025-12-04T09:43:34.1092858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1092938Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1093201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1093268Z layer_outputs = layer_module( 2025-12-04T09:43:34.1093540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1093692Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1093984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1094087Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1094372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1094475Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1094479Z 2025-12-04T09:43:34.1094577Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1094762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1094829Z res = mod(**inputs) 2025-12-04T09:43:34.1095091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1095163Z outputs = self.mobilebert( 2025-12-04T09:43:34.1095428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1095494Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1095766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1095853Z layer_outputs = layer_module( 2025-12-04T09:43:34.1096123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1096205Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1096470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1096544Z self_outputs = self.self( 2025-12-04T09:43:34.1096806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1096871Z self.value(value_tensor) 2025-12-04T09:43:34.1096874Z 2025-12-04T09:43:34.1096977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1097160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1097228Z res = mod(**inputs) 2025-12-04T09:43:34.1097489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1097555Z outputs = self.mobilebert( 2025-12-04T09:43:34.1097821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1097889Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1098158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1098224Z layer_outputs = layer_module( 2025-12-04T09:43:34.1098485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1098644Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1098908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1099010Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1099279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1099354Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1099358Z 2025-12-04T09:43:34.1099458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1099655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1099716Z res = mod(**inputs) 2025-12-04T09:43:34.1099984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1100069Z outputs = self.mobilebert( 2025-12-04T09:43:34.1100363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1100431Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1100690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1100762Z layer_outputs = layer_module( 2025-12-04T09:43:34.1101022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1101168Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1101439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1101556Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1101827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1101907Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1102167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1102259Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1102262Z 2025-12-04T09:43:34.1102359Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1102548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1102610Z res = mod(**inputs) 2025-12-04T09:43:34.1102871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1102947Z outputs = self.mobilebert( 2025-12-04T09:43:34.1103211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1103283Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1103545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1103611Z layer_outputs = layer_module( 2025-12-04T09:43:34.1103881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1103960Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1104225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1104297Z self_outputs = self.self( 2025-12-04T09:43:34.1104562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1104635Z self.query(query_tensor) 2025-12-04T09:43:34.1104639Z 2025-12-04T09:43:34.1104737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1104922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1104989Z res = mod(**inputs) 2025-12-04T09:43:34.1105252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1105324Z outputs = self.mobilebert( 2025-12-04T09:43:34.1105601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1105668Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1105937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1106022Z layer_outputs = layer_module( 2025-12-04T09:43:34.1106306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1106396Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1106663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1106736Z self_outputs = self.self( 2025-12-04T09:43:34.1107003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1107068Z self.key(key_tensor) 2025-12-04T09:43:34.1107071Z 2025-12-04T09:43:34.1107158Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1107234Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1107333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1107542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1107603Z res = mod(**inputs) 2025-12-04T09:43:34.1107873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1107938Z outputs = self.mobilebert( 2025-12-04T09:43:34.1108198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1108272Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1108537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1108610Z layer_outputs = layer_module( 2025-12-04T09:43:34.1108869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1108949Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1109233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1109348Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1109608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1109693Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1109696Z 2025-12-04T09:43:34.1109790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1109981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1110039Z res = mod(**inputs) 2025-12-04T09:43:34.1110297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1110373Z outputs = self.mobilebert( 2025-12-04T09:43:34.1110632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1110705Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1110968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1111033Z layer_outputs = layer_module( 2025-12-04T09:43:34.1111301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1111390Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1111653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1111794Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1112071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1112198Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1112456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1112540Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1112544Z 2025-12-04T09:43:34.1112646Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1112829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1112898Z res = mod(**inputs) 2025-12-04T09:43:34.1113156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1113238Z outputs = self.mobilebert( 2025-12-04T09:43:34.1113524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1113591Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1113873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1113940Z layer_outputs = layer_module( 2025-12-04T09:43:34.1114214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1114309Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1114586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1114691Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1114977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1115055Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1115058Z 2025-12-04T09:43:34.1115158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1115346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1115405Z res = mod(**inputs) 2025-12-04T09:43:34.1115686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1115750Z outputs = self.mobilebert( 2025-12-04T09:43:34.1116029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1116094Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1116373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1116445Z layer_outputs = layer_module( 2025-12-04T09:43:34.1116721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1116807Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1117088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1117191Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1117490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1117597Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1117600Z 2025-12-04T09:43:34.1117715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1117909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1117984Z res = mod(**inputs) 2025-12-04T09:43:34.1118252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1118318Z outputs = self.mobilebert( 2025-12-04T09:43:34.1118579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1118654Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1118913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1118979Z layer_outputs = layer_module( 2025-12-04T09:43:34.1119244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1119350Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1119620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1119739Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1120000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1120086Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1120089Z 2025-12-04T09:43:34.1120185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1120375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1120436Z res = mod(**inputs) 2025-12-04T09:43:34.1120698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1120774Z outputs = self.mobilebert( 2025-12-04T09:43:34.1121037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1121112Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1121376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1121443Z layer_outputs = layer_module( 2025-12-04T09:43:34.1121712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1121800Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1122067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1122197Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1122466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1122590Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1122858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1122947Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1122950Z 2025-12-04T09:43:34.1123056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1123261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1123331Z res = mod(**inputs) 2025-12-04T09:43:34.1123599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1123693Z outputs = self.mobilebert( 2025-12-04T09:43:34.1123988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1124057Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1124327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1124402Z layer_outputs = layer_module( 2025-12-04T09:43:34.1124672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1124768Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1125041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1125152Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1125462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1125545Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1125548Z 2025-12-04T09:43:34.1125655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1125850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1125912Z res = mod(**inputs) 2025-12-04T09:43:34.1126196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1126266Z outputs = self.mobilebert( 2025-12-04T09:43:34.1126544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1126625Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1126920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1127001Z layer_outputs = layer_module( 2025-12-04T09:43:34.1127296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1127392Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1127696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1127812Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1128116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1128234Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1128241Z 2025-12-04T09:43:34.1128348Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1128564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1128631Z res = mod(**inputs) 2025-12-04T09:43:34.1128938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1129006Z outputs = self.mobilebert( 2025-12-04T09:43:34.1129284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1129361Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1129656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1129727Z layer_outputs = layer_module( 2025-12-04T09:43:34.1130011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1130121Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1130422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1130545Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1130821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1130910Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1130913Z 2025-12-04T09:43:34.1131014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1131290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1131363Z res = mod(**inputs) 2025-12-04T09:43:34.1131647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1131746Z outputs = self.mobilebert( 2025-12-04T09:43:34.1132032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1132102Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1132394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1132461Z layer_outputs = layer_module( 2025-12-04T09:43:34.1132753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1132847Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1133128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1133259Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1133541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1133667Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1133954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1134045Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1134049Z 2025-12-04T09:43:34.1134154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1134344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1134412Z res = mod(**inputs) 2025-12-04T09:43:34.1134685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1134758Z outputs = self.mobilebert( 2025-12-04T09:43:34.1135040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1135110Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1135390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1135467Z layer_outputs = layer_module( 2025-12-04T09:43:34.1135744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1135859Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1136134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1136266Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1136563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1136645Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1136648Z 2025-12-04T09:43:34.1136756Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1136956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1137017Z res = mod(**inputs) 2025-12-04T09:43:34.1137306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1137447Z outputs = self.mobilebert( 2025-12-04T09:43:34.1137741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1137915Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1138455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1138555Z layer_outputs = layer_module( 2025-12-04T09:43:34.1138847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1138988Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1139290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1139434Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1139754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1172634Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1172698Z 2025-12-04T09:43:34.1172962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1173203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1173287Z res = mod(**inputs) 2025-12-04T09:43:34.1173619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1173702Z outputs = self.mobilebert( 2025-12-04T09:43:34.1174005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1174084Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1174378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1174456Z layer_outputs = layer_module( 2025-12-04T09:43:34.1174747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1174857Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1175141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1175285Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1175565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1175655Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1175660Z 2025-12-04T09:43:34.1175908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1176121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1176188Z res = mod(**inputs) 2025-12-04T09:43:34.1176535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1176668Z outputs = self.mobilebert( 2025-12-04T09:43:34.1176960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1177036Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1177314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1177393Z layer_outputs = layer_module( 2025-12-04T09:43:34.1177673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1177780Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1178065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1178239Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1178522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1178640Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1178921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1179015Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1179019Z 2025-12-04T09:43:34.1179125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1179335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1179398Z res = mod(**inputs) 2025-12-04T09:43:34.1179671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1179753Z outputs = self.mobilebert( 2025-12-04T09:43:34.1180027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1180108Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1180378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1180443Z layer_outputs = layer_module( 2025-12-04T09:43:34.1180723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1180841Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1181119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1181203Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1181207Z 2025-12-04T09:43:34.1181311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1181514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1181579Z res = mod(**inputs) 2025-12-04T09:43:34.1181850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1181923Z outputs = self.mobilebert( 2025-12-04T09:43:34.1182212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1182290Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1182570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1182656Z layer_outputs = layer_module( 2025-12-04T09:43:34.1182948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1183065Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1183345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1183455Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1183458Z 2025-12-04T09:43:34.1183559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1183761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1183824Z res = mod(**inputs) 2025-12-04T09:43:34.1184093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1184186Z outputs = self.mobilebert( 2025-12-04T09:43:34.1184459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1184536Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1184804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1184871Z layer_outputs = layer_module( 2025-12-04T09:43:34.1185148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1185306Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1185652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1185748Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1185753Z 2025-12-04T09:43:34.1185850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1186048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1186108Z res = mod(**inputs) 2025-12-04T09:43:34.1186386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1186455Z outputs = self.mobilebert( 2025-12-04T09:43:34.1186725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1186805Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1187078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1187144Z layer_outputs = layer_module( 2025-12-04T09:43:34.1187426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1187581Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1187859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1187980Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1188249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1188366Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1188370Z 2025-12-04T09:43:34.1188471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1188671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1188751Z res = mod(**inputs) 2025-12-04T09:43:34.1189038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1189117Z outputs = self.mobilebert( 2025-12-04T09:43:34.1189390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1189460Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1189741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1189807Z layer_outputs = layer_module( 2025-12-04T09:43:34.1190083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1190230Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1190527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1190655Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1190926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1191014Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1191018Z 2025-12-04T09:43:34.1191115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1191300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1191370Z res = mod(**inputs) 2025-12-04T09:43:34.1191639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1191717Z outputs = self.mobilebert( 2025-12-04T09:43:34.1191987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1192058Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1192337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1192404Z layer_outputs = layer_module( 2025-12-04T09:43:34.1192673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1192829Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1193099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1193226Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1193503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1193620Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1193902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1193991Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1193994Z 2025-12-04T09:43:34.1194100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1194288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1194367Z res = mod(**inputs) 2025-12-04T09:43:34.1194648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1194717Z outputs = self.mobilebert( 2025-12-04T09:43:34.1195004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1195098Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1195366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1195442Z layer_outputs = layer_module( 2025-12-04T09:43:34.1195712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1195869Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1196154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1196264Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1196561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1196642Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1196646Z 2025-12-04T09:43:34.1196745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1196939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1197001Z res = mod(**inputs) 2025-12-04T09:43:34.1197278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1197345Z outputs = self.mobilebert( 2025-12-04T09:43:34.1197616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1197690Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1197965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1198033Z layer_outputs = layer_module( 2025-12-04T09:43:34.1198305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1198387Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1198654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1198722Z self_outputs = self.self( 2025-12-04T09:43:34.1198984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1199057Z self.value(value_tensor) 2025-12-04T09:43:34.1199060Z 2025-12-04T09:43:34.1199154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1199347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1199410Z res = mod(**inputs) 2025-12-04T09:43:34.1199670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1199742Z outputs = self.mobilebert( 2025-12-04T09:43:34.1200003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1200070Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1200341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1200421Z layer_outputs = layer_module( 2025-12-04T09:43:34.1200692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1200866Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1201144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1201262Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1201522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1201609Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1201612Z 2025-12-04T09:43:34.1201708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1201899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1201959Z res = mod(**inputs) 2025-12-04T09:43:34.1202220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1202310Z outputs = self.mobilebert( 2025-12-04T09:43:34.1202583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1202657Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1202925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1202988Z layer_outputs = layer_module( 2025-12-04T09:43:34.1203264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1203411Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1203688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1203790Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1204060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1204148Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1204413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1204498Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1204508Z 2025-12-04T09:43:34.1204600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1204782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1204847Z res = mod(**inputs) 2025-12-04T09:43:34.1205122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1205192Z outputs = self.mobilebert( 2025-12-04T09:43:34.1205473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1205541Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1205823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1205889Z layer_outputs = layer_module( 2025-12-04T09:43:34.1206163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1206251Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1206543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1206614Z self_outputs = self.self( 2025-12-04T09:43:34.1206889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1206975Z self.query(query_tensor) 2025-12-04T09:43:34.1206979Z 2025-12-04T09:43:34.1207118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1207312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1207372Z res = mod(**inputs) 2025-12-04T09:43:34.1207651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1207719Z outputs = self.mobilebert( 2025-12-04T09:43:34.1208000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1208068Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1208339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1208432Z layer_outputs = layer_module( 2025-12-04T09:43:34.1208703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1208785Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1209060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1209129Z self_outputs = self.self( 2025-12-04T09:43:34.1209412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1209480Z self.key(key_tensor) 2025-12-04T09:43:34.1209484Z 2025-12-04T09:43:34.1209566Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1209653Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1209758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1209953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1210025Z res = mod(**inputs) 2025-12-04T09:43:34.1210300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1210380Z outputs = self.mobilebert( 2025-12-04T09:43:34.1210654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1210725Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1211011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1211080Z layer_outputs = layer_module( 2025-12-04T09:43:34.1211454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1211553Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1211848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1211986Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1212277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1212365Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1212377Z 2025-12-04T09:43:34.1212485Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1212709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1212787Z res = mod(**inputs) 2025-12-04T09:43:34.1213084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1213172Z outputs = self.mobilebert( 2025-12-04T09:43:34.1213472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1213556Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1213835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1213904Z layer_outputs = layer_module( 2025-12-04T09:43:34.1214175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1214266Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1214542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1214662Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1214965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1215089Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1215374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1215464Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1215467Z 2025-12-04T09:43:34.1215562Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1215765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1215826Z res = mod(**inputs) 2025-12-04T09:43:34.1216108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1216178Z outputs = self.mobilebert( 2025-12-04T09:43:34.1216453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1216530Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1216806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1216879Z layer_outputs = layer_module( 2025-12-04T09:43:34.1217157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1217252Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1217536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1217644Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1217925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1218016Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1218019Z 2025-12-04T09:43:34.1218117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1218319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1218378Z res = mod(**inputs) 2025-12-04T09:43:34.1218653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1218724Z outputs = self.mobilebert( 2025-12-04T09:43:34.1219020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1219096Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1219390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1219476Z layer_outputs = layer_module( 2025-12-04T09:43:34.1219760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1219852Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1220130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1220245Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1220525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1220646Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1220649Z 2025-12-04T09:43:34.1220750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1220962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1221037Z res = mod(**inputs) 2025-12-04T09:43:34.1221314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1221391Z outputs = self.mobilebert( 2025-12-04T09:43:34.1221665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1221733Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1222016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1222083Z layer_outputs = layer_module( 2025-12-04T09:43:34.1222357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1222455Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1222743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1222870Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1223136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1223217Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1223221Z 2025-12-04T09:43:34.1223323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1223511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1223580Z res = mod(**inputs) 2025-12-04T09:43:34.1223846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1223915Z outputs = self.mobilebert( 2025-12-04T09:43:34.1224191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1224258Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1224534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1224600Z layer_outputs = layer_module( 2025-12-04T09:43:34.1224875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1224995Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1225267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1225407Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1225727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1225847Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1226125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1226213Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1226216Z 2025-12-04T09:43:34.1226313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1226508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1226579Z res = mod(**inputs) 2025-12-04T09:43:34.1226848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1226935Z outputs = self.mobilebert( 2025-12-04T09:43:34.1227214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1227281Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1227562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1227628Z layer_outputs = layer_module( 2025-12-04T09:43:34.1227898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1227994Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1228270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1228376Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1228659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1228739Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1228743Z 2025-12-04T09:43:34.1228846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1229033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1229091Z res = mod(**inputs) 2025-12-04T09:43:34.1229371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1229440Z outputs = self.mobilebert( 2025-12-04T09:43:34.1229730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1229800Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1230091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1230168Z layer_outputs = layer_module( 2025-12-04T09:43:34.1230440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1230529Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1230808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1230912Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1231207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1231316Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1231337Z 2025-12-04T09:43:34.1231435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1231648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1231712Z res = mod(**inputs) 2025-12-04T09:43:34.1231992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1232058Z outputs = self.mobilebert( 2025-12-04T09:43:34.1232332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1232406Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1232685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1232753Z layer_outputs = layer_module( 2025-12-04T09:43:34.1233041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1233149Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1233430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1233548Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1233823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1233909Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1233913Z 2025-12-04T09:43:34.1234016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1234214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1234277Z res = mod(**inputs) 2025-12-04T09:43:34.1234557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1234634Z outputs = self.mobilebert( 2025-12-04T09:43:34.1234915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1234985Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1235266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1235334Z layer_outputs = layer_module( 2025-12-04T09:43:34.1235631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1235721Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1235997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1236132Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1236411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1236537Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1236816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1236907Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1236911Z 2025-12-04T09:43:34.1237019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1237234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1237308Z res = mod(**inputs) 2025-12-04T09:43:34.1237590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1237680Z outputs = self.mobilebert( 2025-12-04T09:43:34.1237988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1238062Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1238339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1238415Z layer_outputs = layer_module( 2025-12-04T09:43:34.1238694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1238794Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1239071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1239202Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1239498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1239583Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1239586Z 2025-12-04T09:43:34.1239696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1239889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1239955Z res = mod(**inputs) 2025-12-04T09:43:34.1240240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1240314Z outputs = self.mobilebert( 2025-12-04T09:43:34.1240595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1240681Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1240959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1241039Z layer_outputs = layer_module( 2025-12-04T09:43:34.1241316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1241406Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1241694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1241803Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1242092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1242203Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1242209Z 2025-12-04T09:43:34.1242313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1242516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1242582Z res = mod(**inputs) 2025-12-04T09:43:34.1242864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1242937Z outputs = self.mobilebert( 2025-12-04T09:43:34.1243216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1243295Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1243586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1243656Z layer_outputs = layer_module( 2025-12-04T09:43:34.1243961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1244069Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1244361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1244489Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1244783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1244875Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1244878Z 2025-12-04T09:43:34.1244985Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1245194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1245260Z res = mod(**inputs) 2025-12-04T09:43:34.1245579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1245660Z outputs = self.mobilebert( 2025-12-04T09:43:34.1245952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1246025Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1246331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1246402Z layer_outputs = layer_module( 2025-12-04T09:43:34.1246703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1246797Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1247093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1247234Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1247533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1247667Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1247958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1248052Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1248056Z 2025-12-04T09:43:34.1248165Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1248361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1248425Z res = mod(**inputs) 2025-12-04T09:43:34.1248723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1248795Z outputs = self.mobilebert( 2025-12-04T09:43:34.1249093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1249164Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1249455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1249530Z layer_outputs = layer_module( 2025-12-04T09:43:34.1249836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1249966Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1250260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1250365Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1250369Z 2025-12-04T09:43:34.1250495Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1250699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1250768Z res = mod(**inputs) 2025-12-04T09:43:34.1251118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1251272Z outputs = self.mobilebert( 2025-12-04T09:43:34.1251617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1251698Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1252018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1252142Z layer_outputs = layer_module( 2025-12-04T09:43:34.1252457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1252593Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1252901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1253017Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1253021Z 2025-12-04T09:43:34.1253138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1253344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1253420Z res = mod(**inputs) 2025-12-04T09:43:34.1253712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1253790Z outputs = self.mobilebert( 2025-12-04T09:43:34.1254095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1254171Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1254467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1254550Z layer_outputs = layer_module( 2025-12-04T09:43:34.1254844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1255018Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1255309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1255407Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1255412Z 2025-12-04T09:43:34.1255522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1255725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1255794Z res = mod(**inputs) 2025-12-04T09:43:34.1256089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1256161Z outputs = self.mobilebert( 2025-12-04T09:43:34.1256462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1256556Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1256854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1256937Z layer_outputs = layer_module( 2025-12-04T09:43:34.1257251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1257441Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1257736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1257862Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1258160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1258257Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1258261Z 2025-12-04T09:43:34.1258373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1258577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1258663Z res = mod(**inputs) 2025-12-04T09:43:34.1258969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1259045Z outputs = self.mobilebert( 2025-12-04T09:43:34.1259348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1259423Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1259719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1259799Z layer_outputs = layer_module( 2025-12-04T09:43:34.1260099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1260263Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1260571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1260701Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1261006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1261093Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1261096Z 2025-12-04T09:43:34.1261211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1261411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1261475Z res = mod(**inputs) 2025-12-04T09:43:34.1261760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1261829Z outputs = self.mobilebert( 2025-12-04T09:43:34.1262126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1262210Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1262506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1262578Z layer_outputs = layer_module( 2025-12-04T09:43:34.1262886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1263048Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1263368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1263487Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1263783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1263927Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1264204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1264298Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1264302Z 2025-12-04T09:43:34.1264415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1264613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1264677Z res = mod(**inputs) 2025-12-04T09:43:34.1264950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1265025Z outputs = self.mobilebert( 2025-12-04T09:43:34.1265321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1265400Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1265680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1265747Z layer_outputs = layer_module( 2025-12-04T09:43:34.1266046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1266203Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1266481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1266595Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1266872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1266966Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1266970Z 2025-12-04T09:43:34.1267069Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1267261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1267332Z res = mod(**inputs) 2025-12-04T09:43:34.1267606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1267682Z outputs = self.mobilebert( 2025-12-04T09:43:34.1267958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1268028Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1268310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1268382Z layer_outputs = layer_module( 2025-12-04T09:43:34.1268659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1268752Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1269029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1269106Z self_outputs = self.self( 2025-12-04T09:43:34.1269401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1269473Z self.value(value_tensor) 2025-12-04T09:43:34.1269477Z 2025-12-04T09:43:34.1269585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1269798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1269868Z res = mod(**inputs) 2025-12-04T09:43:34.1270169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1270239Z outputs = self.mobilebert( 2025-12-04T09:43:34.1270523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1270593Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1270868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1270947Z layer_outputs = layer_module( 2025-12-04T09:43:34.1271222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1271389Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1271686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1271793Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1272078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1272157Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1272161Z 2025-12-04T09:43:34.1272400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1272611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1272676Z res = mod(**inputs) 2025-12-04T09:43:34.1272975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1273054Z outputs = self.mobilebert( 2025-12-04T09:43:34.1273347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1273432Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1273722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1273805Z layer_outputs = layer_module( 2025-12-04T09:43:34.1274096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1274264Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1274568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1274685Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1274994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1275086Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1275378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1275483Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1275487Z 2025-12-04T09:43:34.1275592Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1275805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1275929Z res = mod(**inputs) 2025-12-04T09:43:34.1276207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1276314Z outputs = self.mobilebert( 2025-12-04T09:43:34.1276592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1276685Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1276970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1277041Z layer_outputs = layer_module( 2025-12-04T09:43:34.1277325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1277410Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1277687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1277766Z self_outputs = self.self( 2025-12-04T09:43:34.1278044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1278149Z self.query(query_tensor) 2025-12-04T09:43:34.1278152Z 2025-12-04T09:43:34.1278260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1278470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1278543Z res = mod(**inputs) 2025-12-04T09:43:34.1278848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1278921Z outputs = self.mobilebert( 2025-12-04T09:43:34.1279236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1279312Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1279625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1279701Z layer_outputs = layer_module( 2025-12-04T09:43:34.1280007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1280103Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1280409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1280491Z self_outputs = self.self( 2025-12-04T09:43:34.1280801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1280874Z self.key(key_tensor) 2025-12-04T09:43:34.1280877Z 2025-12-04T09:43:34.1280971Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1281053Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1281161Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1281381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1281449Z res = mod(**inputs) 2025-12-04T09:43:34.1281760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1281832Z outputs = self.mobilebert( 2025-12-04T09:43:34.1282137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1282219Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1282541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1282616Z layer_outputs = layer_module( 2025-12-04T09:43:34.1282914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1283020Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1283616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1283754Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1284054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1284153Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1284157Z 2025-12-04T09:43:34.1284271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1284488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1284558Z res = mod(**inputs) 2025-12-04T09:43:34.1284855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1284958Z outputs = self.mobilebert( 2025-12-04T09:43:34.1285256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1285330Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1285634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1285707Z layer_outputs = layer_module( 2025-12-04T09:43:34.1286016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1286103Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1286396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1286535Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1286834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1286971Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1287268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1287363Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1287367Z 2025-12-04T09:43:34.1287480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1287687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1287759Z res = mod(**inputs) 2025-12-04T09:43:34.1288054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1288130Z outputs = self.mobilebert( 2025-12-04T09:43:34.1288436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1288509Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1288810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1288887Z layer_outputs = layer_module( 2025-12-04T09:43:34.1289182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1289286Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1289597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1289717Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1290039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1290144Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1290148Z 2025-12-04T09:43:34.1290263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1290469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1290534Z res = mod(**inputs) 2025-12-04T09:43:34.1290835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1290905Z outputs = self.mobilebert( 2025-12-04T09:43:34.1291271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1291367Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1291673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1291778Z layer_outputs = layer_module( 2025-12-04T09:43:34.1292085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1292185Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1292489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1292608Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1292916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1293034Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1293038Z 2025-12-04T09:43:34.1293148Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1293369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1293438Z res = mod(**inputs) 2025-12-04T09:43:34.1293738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1293825Z outputs = self.mobilebert( 2025-12-04T09:43:34.1294128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1294210Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1294504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1294577Z layer_outputs = layer_module( 2025-12-04T09:43:34.1294875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1294975Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1295276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1295407Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1295698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1295791Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1295795Z 2025-12-04T09:43:34.1295898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1296129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1296198Z res = mod(**inputs) 2025-12-04T09:43:34.1296491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1296601Z outputs = self.mobilebert( 2025-12-04T09:43:34.1296907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1296984Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1297286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1297359Z layer_outputs = layer_module( 2025-12-04T09:43:34.1297656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1297752Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1298052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1298208Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1298503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1298637Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1298932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1299028Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1299031Z 2025-12-04T09:43:34.1299141Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1299345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1299411Z res = mod(**inputs) 2025-12-04T09:43:34.1299706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1299781Z outputs = self.mobilebert( 2025-12-04T09:43:34.1300080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1300154Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1300444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1300522Z layer_outputs = layer_module( 2025-12-04T09:43:34.1300811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1300914Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1301206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1301320Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1301619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1301707Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1301710Z 2025-12-04T09:43:34.1301820Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1302021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1302085Z res = mod(**inputs) 2025-12-04T09:43:34.1302379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1302468Z outputs = self.mobilebert( 2025-12-04T09:43:34.1302761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1302842Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1303151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1303250Z layer_outputs = layer_module( 2025-12-04T09:43:34.1303541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1303637Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1303935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1304048Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1304349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1304464Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1304468Z 2025-12-04T09:43:34.1304594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1304808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1304873Z res = mod(**inputs) 2025-12-04T09:43:34.1305163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1305243Z outputs = self.mobilebert( 2025-12-04T09:43:34.1305534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1305611Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1305905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1305976Z layer_outputs = layer_module( 2025-12-04T09:43:34.1306274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1306372Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1306670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1306797Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1307089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1307181Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1307184Z 2025-12-04T09:43:34.1307290Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1307493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1307565Z res = mod(**inputs) 2025-12-04T09:43:34.1307857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1307939Z outputs = self.mobilebert( 2025-12-04T09:43:34.1308233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1308306Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1308607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1308678Z layer_outputs = layer_module( 2025-12-04T09:43:34.1308994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1309092Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1309385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1309541Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1309851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1309980Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1310302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1310391Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1310394Z 2025-12-04T09:43:34.1310500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1310694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1310756Z res = mod(**inputs) 2025-12-04T09:43:34.1311056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1311149Z outputs = self.mobilebert( 2025-12-04T09:43:34.1311458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1311534Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1311836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1311916Z layer_outputs = layer_module( 2025-12-04T09:43:34.1312216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1312319Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1312625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1312743Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1313053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1313140Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1313143Z 2025-12-04T09:43:34.1313249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1313466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1313531Z res = mod(**inputs) 2025-12-04T09:43:34.1313835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1313910Z outputs = self.mobilebert( 2025-12-04T09:43:34.1314208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1314294Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1314594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1314673Z layer_outputs = layer_module( 2025-12-04T09:43:34.1314973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1315067Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1315371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1315485Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1315805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1315929Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1315968Z 2025-12-04T09:43:34.1316078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1316311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1316380Z res = mod(**inputs) 2025-12-04T09:43:34.1316677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1316760Z outputs = self.mobilebert( 2025-12-04T09:43:34.1317061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1317140Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1317420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1317490Z layer_outputs = layer_module( 2025-12-04T09:43:34.1317797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1317913Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1318207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1318345Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1318638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1318732Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1318735Z 2025-12-04T09:43:34.1318843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1319047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1319120Z res = mod(**inputs) 2025-12-04T09:43:34.1319418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1319500Z outputs = self.mobilebert( 2025-12-04T09:43:34.1319793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1319867Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1320163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1320236Z layer_outputs = layer_module( 2025-12-04T09:43:34.1320529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1320632Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1320922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1321059Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1321350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1321475Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1321772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1321867Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1321870Z 2025-12-04T09:43:34.1321982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1322210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1322277Z res = mod(**inputs) 2025-12-04T09:43:34.1322575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1322669Z outputs = self.mobilebert( 2025-12-04T09:43:34.1322982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1323058Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1323351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1323428Z layer_outputs = layer_module( 2025-12-04T09:43:34.1323726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1323851Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1324148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1324252Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1324256Z 2025-12-04T09:43:34.1324372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1324577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1324643Z res = mod(**inputs) 2025-12-04T09:43:34.1324945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1325018Z outputs = self.mobilebert( 2025-12-04T09:43:34.1325317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1325391Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1325679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1325762Z layer_outputs = layer_module( 2025-12-04T09:43:34.1326053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1326174Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1326471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1326584Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1326588Z 2025-12-04T09:43:34.1326700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1326904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1326968Z res = mod(**inputs) 2025-12-04T09:43:34.1327267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1327343Z outputs = self.mobilebert( 2025-12-04T09:43:34.1327643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1327716Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1328007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1328086Z layer_outputs = layer_module( 2025-12-04T09:43:34.1328375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1328561Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1328853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1328992Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1328998Z 2025-12-04T09:43:34.1329113Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1329336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1329404Z res = mod(**inputs) 2025-12-04T09:43:34.1329703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1329775Z outputs = self.mobilebert( 2025-12-04T09:43:34.1330074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1330146Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1330433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1330516Z layer_outputs = layer_module( 2025-12-04T09:43:34.1330831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1331002Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1331365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1331504Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1331820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1331921Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1331929Z 2025-12-04T09:43:34.1332047Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1332272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1332350Z res = mod(**inputs) 2025-12-04T09:43:34.1332672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1332758Z outputs = self.mobilebert( 2025-12-04T09:43:34.1333051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1333137Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1333432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1333514Z layer_outputs = layer_module( 2025-12-04T09:43:34.1333809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1333971Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1334274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1334404Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1334715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1334804Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1334807Z 2025-12-04T09:43:34.1334915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1335131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1335219Z res = mod(**inputs) 2025-12-04T09:43:34.1335514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1335595Z outputs = self.mobilebert( 2025-12-04T09:43:34.1335912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1336014Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1336307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1336379Z layer_outputs = layer_module( 2025-12-04T09:43:34.1336738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1336897Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1337200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1337327Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1337644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1337783Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1338083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1338186Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1338190Z 2025-12-04T09:43:34.1338298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1338504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1338576Z res = mod(**inputs) 2025-12-04T09:43:34.1338872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1338944Z outputs = self.mobilebert( 2025-12-04T09:43:34.1339250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1339327Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1339627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1339698Z layer_outputs = layer_module( 2025-12-04T09:43:34.1339990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1340164Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1340455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1340564Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1340835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1340916Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1340920Z 2025-12-04T09:43:34.1341025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1341213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1341272Z res = mod(**inputs) 2025-12-04T09:43:34.1341551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1341617Z outputs = self.mobilebert( 2025-12-04T09:43:34.1341911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1341983Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1342259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1342353Z layer_outputs = layer_module( 2025-12-04T09:43:34.1342647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1342740Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1343015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1343085Z self_outputs = self.self( 2025-12-04T09:43:34.1343381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1343450Z self.value(value_tensor) 2025-12-04T09:43:34.1343453Z 2025-12-04T09:43:34.1343551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1343747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1343834Z res = mod(**inputs) 2025-12-04T09:43:34.1344117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1344185Z outputs = self.mobilebert( 2025-12-04T09:43:34.1344458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1344535Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1344807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1344886Z layer_outputs = layer_module( 2025-12-04T09:43:34.1345160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1345320Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1345606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1345717Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1345996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1346083Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1346086Z 2025-12-04T09:43:34.1346188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1346388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1346450Z res = mod(**inputs) 2025-12-04T09:43:34.1346725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1346805Z outputs = self.mobilebert( 2025-12-04T09:43:34.1347080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1347158Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1347432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1347502Z layer_outputs = layer_module( 2025-12-04T09:43:34.1347782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1347955Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1348240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1348363Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1348659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1348753Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1349032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1349122Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1349134Z 2025-12-04T09:43:34.1349232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1349424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1349496Z res = mod(**inputs) 2025-12-04T09:43:34.1349771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1349843Z outputs = self.mobilebert( 2025-12-04T09:43:34.1350144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1350225Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1350502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1350568Z layer_outputs = layer_module( 2025-12-04T09:43:34.1350840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1350925Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1351199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1351266Z self_outputs = self.self( 2025-12-04T09:43:34.1351540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1351610Z self.query(query_tensor) 2025-12-04T09:43:34.1351613Z 2025-12-04T09:43:34.1351719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1351906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1351966Z res = mod(**inputs) 2025-12-04T09:43:34.1352240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1352306Z outputs = self.mobilebert( 2025-12-04T09:43:34.1352583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1352651Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1352919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1352997Z layer_outputs = layer_module( 2025-12-04T09:43:34.1353271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1353351Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1353629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1353694Z self_outputs = self.self( 2025-12-04T09:43:34.1353973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1354057Z self.key(key_tensor) 2025-12-04T09:43:34.1354061Z 2025-12-04T09:43:34.1354146Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1354233Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1354336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1354559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1354627Z res = mod(**inputs) 2025-12-04T09:43:34.1354914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1354991Z outputs = self.mobilebert( 2025-12-04T09:43:34.1355265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1355334Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1355615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1355684Z layer_outputs = layer_module( 2025-12-04T09:43:34.1355966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1356065Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1356344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1356475Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1356752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1356835Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1356846Z 2025-12-04T09:43:34.1356944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1357136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1357203Z res = mod(**inputs) 2025-12-04T09:43:34.1357480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1357551Z outputs = self.mobilebert( 2025-12-04T09:43:34.1357836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1357905Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1358188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1358258Z layer_outputs = layer_module( 2025-12-04T09:43:34.1358534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1358625Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1358899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1359025Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1359313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1359438Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1359725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1359816Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1359819Z 2025-12-04T09:43:34.1359919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1360136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1360199Z res = mod(**inputs) 2025-12-04T09:43:34.1360484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1360574Z outputs = self.mobilebert( 2025-12-04T09:43:34.1360865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1360945Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1361222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1361298Z layer_outputs = layer_module( 2025-12-04T09:43:34.1361575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1361668Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1361956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1362067Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1362362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1362454Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1362457Z 2025-12-04T09:43:34.1362557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1362754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1362817Z res = mod(**inputs) 2025-12-04T09:43:34.1363090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1363165Z outputs = self.mobilebert( 2025-12-04T09:43:34.1363440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1363518Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1363793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1363864Z layer_outputs = layer_module( 2025-12-04T09:43:34.1364147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1364237Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1364509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1364623Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1364899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1365016Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1365019Z 2025-12-04T09:43:34.1365120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1365310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1365381Z res = mod(**inputs) 2025-12-04T09:43:34.1365652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1365726Z outputs = self.mobilebert( 2025-12-04T09:43:34.1365999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1366068Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1366367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1366439Z layer_outputs = layer_module( 2025-12-04T09:43:34.1366713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1366831Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1367137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1367270Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1367547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1367633Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1367637Z 2025-12-04T09:43:34.1367750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1367956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1368029Z res = mod(**inputs) 2025-12-04T09:43:34.1368322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1368413Z outputs = self.mobilebert( 2025-12-04T09:43:34.1368717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1368790Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1369088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1369160Z layer_outputs = layer_module( 2025-12-04T09:43:34.1369453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1369557Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1369850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1369982Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1370286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1370411Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1370710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1370804Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1370808Z 2025-12-04T09:43:34.1370911Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1371130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1371285Z res = mod(**inputs) 2025-12-04T09:43:34.1371611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1371693Z outputs = self.mobilebert( 2025-12-04T09:43:34.1371997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1372085Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1372517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1372597Z layer_outputs = layer_module( 2025-12-04T09:43:34.1372899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1373043Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1373346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1373462Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1373785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1373905Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1373910Z 2025-12-04T09:43:34.1374019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1374237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1374308Z res = mod(**inputs) 2025-12-04T09:43:34.1374605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1374691Z outputs = self.mobilebert( 2025-12-04T09:43:34.1374988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1375066Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1375395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1375471Z layer_outputs = layer_module( 2025-12-04T09:43:34.1375773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1375866Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1376157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1376278Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1376569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1376692Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1376698Z 2025-12-04T09:43:34.1376805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1377010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1377085Z res = mod(**inputs) 2025-12-04T09:43:34.1377376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1377455Z outputs = self.mobilebert( 2025-12-04T09:43:34.1377748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1377823Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1378125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1378200Z layer_outputs = layer_module( 2025-12-04T09:43:34.1378491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1378598Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1378893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1379033Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1379322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1379407Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1379410Z 2025-12-04T09:43:34.1379524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1379749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1379824Z res = mod(**inputs) 2025-12-04T09:43:34.1380118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1380213Z outputs = self.mobilebert( 2025-12-04T09:43:34.1380536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1380612Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1380907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1380986Z layer_outputs = layer_module( 2025-12-04T09:43:34.1381282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1381384Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1381677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1381824Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1382128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1382253Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1382552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1382646Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1382649Z 2025-12-04T09:43:34.1382754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1382966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1383032Z res = mod(**inputs) 2025-12-04T09:43:34.1383331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1383408Z outputs = self.mobilebert( 2025-12-04T09:43:34.1383701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1383784Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1384079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1384151Z layer_outputs = layer_module( 2025-12-04T09:43:34.1384450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1384548Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1384853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1384969Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1385261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1385352Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1385356Z 2025-12-04T09:43:34.1385463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1385672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1385739Z res = mod(**inputs) 2025-12-04T09:43:34.1386031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1386133Z outputs = self.mobilebert( 2025-12-04T09:43:34.1386439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1386531Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1386847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1386941Z layer_outputs = layer_module( 2025-12-04T09:43:34.1387242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1387336Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1387629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1387750Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1388043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1388165Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1388187Z 2025-12-04T09:43:34.1388295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1388503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1388577Z res = mod(**inputs) 2025-12-04T09:43:34.1388869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1388941Z outputs = self.mobilebert( 2025-12-04T09:43:34.1389242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1389315Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1389614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1389686Z layer_outputs = layer_module( 2025-12-04T09:43:34.1389979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1390084Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1390379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1390511Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1390800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1390887Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1390891Z 2025-12-04T09:43:34.1391004Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1391207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1391272Z res = mod(**inputs) 2025-12-04T09:43:34.1391573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1391645Z outputs = self.mobilebert( 2025-12-04T09:43:34.1391942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1392015Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1392305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1392384Z layer_outputs = layer_module( 2025-12-04T09:43:34.1392694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1392798Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1393092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1393247Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1393564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1393692Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1393999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1394094Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1394097Z 2025-12-04T09:43:34.1394205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1394417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1394484Z res = mod(**inputs) 2025-12-04T09:43:34.1394778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1394882Z outputs = self.mobilebert( 2025-12-04T09:43:34.1395179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1395263Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1395560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1395633Z layer_outputs = layer_module( 2025-12-04T09:43:34.1395936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1396065Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1396372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1396457Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1396461Z 2025-12-04T09:43:34.1396564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1396771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1396834Z res = mod(**inputs) 2025-12-04T09:43:34.1397113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1397193Z outputs = self.mobilebert( 2025-12-04T09:43:34.1397474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1397555Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1397834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1397905Z layer_outputs = layer_module( 2025-12-04T09:43:34.1398195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1398313Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1398601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1398712Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1398716Z 2025-12-04T09:43:34.1398823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1399040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1399104Z res = mod(**inputs) 2025-12-04T09:43:34.1399384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1399477Z outputs = self.mobilebert( 2025-12-04T09:43:34.1399781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1399861Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1400143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1400212Z layer_outputs = layer_module( 2025-12-04T09:43:34.1400510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1400669Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1400959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1401055Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1401076Z 2025-12-04T09:43:34.1401178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1401379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1401442Z res = mod(**inputs) 2025-12-04T09:43:34.1401735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1401804Z outputs = self.mobilebert( 2025-12-04T09:43:34.1402090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1402168Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1402454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1402522Z layer_outputs = layer_module( 2025-12-04T09:43:34.1402820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1402977Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1403270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1403392Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1403676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1403773Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1403779Z 2025-12-04T09:43:34.1403878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1404082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1404149Z res = mod(**inputs) 2025-12-04T09:43:34.1404440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1404516Z outputs = self.mobilebert( 2025-12-04T09:43:34.1404802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1404871Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1405162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1405229Z layer_outputs = layer_module( 2025-12-04T09:43:34.1405546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1405711Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1406031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1406184Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1406477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1406568Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1406572Z 2025-12-04T09:43:34.1406676Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1406878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1406950Z res = mod(**inputs) 2025-12-04T09:43:34.1407240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1407317Z outputs = self.mobilebert( 2025-12-04T09:43:34.1407625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1407700Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1407995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1408066Z layer_outputs = layer_module( 2025-12-04T09:43:34.1408358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1408525Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1408818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1408952Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1409245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1409371Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1409669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1409762Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1409766Z 2025-12-04T09:43:34.1409876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1410078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1410145Z res = mod(**inputs) 2025-12-04T09:43:34.1410442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1410515Z outputs = self.mobilebert( 2025-12-04T09:43:34.1410808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1410892Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1411180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1411332Z layer_outputs = layer_module( 2025-12-04T09:43:34.1411639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1411815Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1412156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1412278Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1412609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1412700Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1412721Z 2025-12-04T09:43:34.1412832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1413057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1413134Z res = mod(**inputs) 2025-12-04T09:43:34.1413426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1413502Z outputs = self.mobilebert( 2025-12-04T09:43:34.1413800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1413888Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1414203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1414300Z layer_outputs = layer_module( 2025-12-04T09:43:34.1414622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1414712Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1415028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1415107Z self_outputs = self.self( 2025-12-04T09:43:34.1415415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1415502Z self.value(value_tensor) 2025-12-04T09:43:34.1415506Z 2025-12-04T09:43:34.1415615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1415831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1415903Z res = mod(**inputs) 2025-12-04T09:43:34.1416215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1416295Z outputs = self.mobilebert( 2025-12-04T09:43:34.1416602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1416677Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1416999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1417075Z layer_outputs = layer_module( 2025-12-04T09:43:34.1417396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1417566Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1417883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1418007Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1418320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1418414Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1418418Z 2025-12-04T09:43:34.1418527Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1418763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1418841Z res = mod(**inputs) 2025-12-04T09:43:34.1419151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1419249Z outputs = self.mobilebert( 2025-12-04T09:43:34.1419587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1419667Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1419988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1420061Z layer_outputs = layer_module( 2025-12-04T09:43:34.1420379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1420558Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1420871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1420995Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1421330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1421425Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1421737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1421834Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1421838Z 2025-12-04T09:43:34.1421952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1422163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1422230Z res = mod(**inputs) 2025-12-04T09:43:34.1422547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1422622Z outputs = self.mobilebert( 2025-12-04T09:43:34.1422929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1423012Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1423324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1423405Z layer_outputs = layer_module( 2025-12-04T09:43:34.1423709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1423797Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1424111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1424185Z self_outputs = self.self( 2025-12-04T09:43:34.1424493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1424571Z self.query(query_tensor) 2025-12-04T09:43:34.1424575Z 2025-12-04T09:43:34.1424686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1424910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1424974Z res = mod(**inputs) 2025-12-04T09:43:34.1425268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1425346Z outputs = self.mobilebert( 2025-12-04T09:43:34.1425666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1425749Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1426045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1426137Z layer_outputs = layer_module( 2025-12-04T09:43:34.1426460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1426550Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1426850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1426921Z self_outputs = self.self( 2025-12-04T09:43:34.1427212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1427289Z self.key(key_tensor) 2025-12-04T09:43:34.1427292Z 2025-12-04T09:43:34.1427378Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1427461Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1427578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1427804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1427877Z res = mod(**inputs) 2025-12-04T09:43:34.1428173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1428247Z outputs = self.mobilebert( 2025-12-04T09:43:34.1428546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1428618Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1428914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1428993Z layer_outputs = layer_module( 2025-12-04T09:43:34.1429280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1429376Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1429666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1429792Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1430090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1430176Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1430179Z 2025-12-04T09:43:34.1430291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1430498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1430563Z res = mod(**inputs) 2025-12-04T09:43:34.1430860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1430934Z outputs = self.mobilebert( 2025-12-04T09:43:34.1431225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1431308Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1431602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1431680Z layer_outputs = layer_module( 2025-12-04T09:43:34.1431973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1432112Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1432413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1432536Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1432872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1433005Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1433295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1433391Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1433394Z 2025-12-04T09:43:34.1433494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1433695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1433763Z res = mod(**inputs) 2025-12-04T09:43:34.1434057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1434156Z outputs = self.mobilebert( 2025-12-04T09:43:34.1434451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1434522Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1434809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1434877Z layer_outputs = layer_module( 2025-12-04T09:43:34.1435159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1435252Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1435530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1435649Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1435929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1436017Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1436021Z 2025-12-04T09:43:34.1436121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1436312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1436381Z res = mod(**inputs) 2025-12-04T09:43:34.1436656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1436723Z outputs = self.mobilebert( 2025-12-04T09:43:34.1437006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1437074Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1437357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1437425Z layer_outputs = layer_module( 2025-12-04T09:43:34.1437703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1437801Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1438077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1438195Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1438485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1438596Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1438600Z 2025-12-04T09:43:34.1438723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1438919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1438997Z res = mod(**inputs) 2025-12-04T09:43:34.1439281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1439349Z outputs = self.mobilebert( 2025-12-04T09:43:34.1439630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1439698Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1439972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1440048Z layer_outputs = layer_module( 2025-12-04T09:43:34.1440321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1440447Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1440728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1440853Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1441138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1441221Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1441224Z 2025-12-04T09:43:34.1441330Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1441535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1441597Z res = mod(**inputs) 2025-12-04T09:43:34.1441891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1441962Z outputs = self.mobilebert( 2025-12-04T09:43:34.1442239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1442318Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1442596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1442670Z layer_outputs = layer_module( 2025-12-04T09:43:34.1442947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1443039Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1443325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1443447Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1443729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1443848Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1444125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1444222Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1444226Z 2025-12-04T09:43:34.1444326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1444538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1444611Z res = mod(**inputs) 2025-12-04T09:43:34.1444890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1444985Z outputs = self.mobilebert( 2025-12-04T09:43:34.1445277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1445349Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1445639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1445708Z layer_outputs = layer_module( 2025-12-04T09:43:34.1446001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1446093Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1446376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1446494Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1446787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1446875Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1446885Z 2025-12-04T09:43:34.1446991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1447194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1447267Z res = mod(**inputs) 2025-12-04T09:43:34.1447560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1447631Z outputs = self.mobilebert( 2025-12-04T09:43:34.1447934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1448008Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1448309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1448384Z layer_outputs = layer_module( 2025-12-04T09:43:34.1448676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1448780Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1449071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1449184Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1449487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1449602Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1449608Z 2025-12-04T09:43:34.1449723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1449927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1449996Z res = mod(**inputs) 2025-12-04T09:43:34.1450297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1450370Z outputs = self.mobilebert( 2025-12-04T09:43:34.1450667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1450739Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1451050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1451136Z layer_outputs = layer_module( 2025-12-04T09:43:34.1451526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1451667Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1451991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1452129Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1452439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1452527Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1452531Z 2025-12-04T09:43:34.1452639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1452866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1452944Z res = mod(**inputs) 2025-12-04T09:43:34.1453242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1453338Z outputs = self.mobilebert( 2025-12-04T09:43:34.1453632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1453717Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1454013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1454095Z layer_outputs = layer_module( 2025-12-04T09:43:34.1454388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1454486Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1454785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1454915Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1455207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1455338Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1455633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1455736Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1455739Z 2025-12-04T09:43:34.1455844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1456050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1456124Z res = mod(**inputs) 2025-12-04T09:43:34.1456418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1456500Z outputs = self.mobilebert( 2025-12-04T09:43:34.1456801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1456876Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1457175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1457247Z layer_outputs = layer_module( 2025-12-04T09:43:34.1457539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1457664Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1457957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1458098Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1458412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1458499Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1458503Z 2025-12-04T09:43:34.1458616Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1458820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1458893Z res = mod(**inputs) 2025-12-04T09:43:34.1459184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1459257Z outputs = self.mobilebert( 2025-12-04T09:43:34.1459559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1459631Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1459932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1460003Z layer_outputs = layer_module( 2025-12-04T09:43:34.1460278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1460374Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1460648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1460755Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1461037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1461147Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1461153Z 2025-12-04T09:43:34.1461259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1461454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1461517Z res = mod(**inputs) 2025-12-04T09:43:34.1461799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1461869Z outputs = self.mobilebert( 2025-12-04T09:43:34.1462154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1462224Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1462496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1462574Z layer_outputs = layer_module( 2025-12-04T09:43:34.1462848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1462942Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1463226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1463348Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1463632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1463714Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1463717Z 2025-12-04T09:43:34.1463831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1464034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1464096Z res = mod(**inputs) 2025-12-04T09:43:34.1464397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1464466Z outputs = self.mobilebert( 2025-12-04T09:43:34.1464766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1464846Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1465136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1465204Z layer_outputs = layer_module( 2025-12-04T09:43:34.1465490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1465581Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1465866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1466007Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1466290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1466417Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1466692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1466787Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1466791Z 2025-12-04T09:43:34.1466892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1467083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1467153Z res = mod(**inputs) 2025-12-04T09:43:34.1467429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1467508Z outputs = self.mobilebert( 2025-12-04T09:43:34.1467786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1467854Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1468136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1468207Z layer_outputs = layer_module( 2025-12-04T09:43:34.1468484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1468609Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1468887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1468977Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1468980Z 2025-12-04T09:43:34.1469078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1469269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1469340Z res = mod(**inputs) 2025-12-04T09:43:34.1469612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1469688Z outputs = self.mobilebert( 2025-12-04T09:43:34.1469961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1470047Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1470335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1470422Z layer_outputs = layer_module( 2025-12-04T09:43:34.1470718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1470844Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1471116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1471230Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1471233Z 2025-12-04T09:43:34.1471334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1471527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1471599Z res = mod(**inputs) 2025-12-04T09:43:34.1471885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1471978Z outputs = self.mobilebert( 2025-12-04T09:43:34.1472246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1472434Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1472713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1472781Z layer_outputs = layer_module( 2025-12-04T09:43:34.1473055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1473219Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1473490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1473589Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1473596Z 2025-12-04T09:43:34.1473694Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1473882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1473953Z res = mod(**inputs) 2025-12-04T09:43:34.1474226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1474302Z outputs = self.mobilebert( 2025-12-04T09:43:34.1474571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1474640Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1474919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1474988Z layer_outputs = layer_module( 2025-12-04T09:43:34.1475270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1475424Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1475698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1475825Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1476097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1476185Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1476239Z 2025-12-04T09:43:34.1476339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1476527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1476625Z res = mod(**inputs) 2025-12-04T09:43:34.1476916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1476984Z outputs = self.mobilebert( 2025-12-04T09:43:34.1477263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1477331Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1477606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1477675Z layer_outputs = layer_module( 2025-12-04T09:43:34.1477946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1478102Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1478393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1478513Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1478785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1478863Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1478867Z 2025-12-04T09:43:34.1478969Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1479155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1479216Z res = mod(**inputs) 2025-12-04T09:43:34.1479492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1479559Z outputs = self.mobilebert( 2025-12-04T09:43:34.1479835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1479905Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1480170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1480245Z layer_outputs = layer_module( 2025-12-04T09:43:34.1480511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1480664Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1480934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1481050Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1481322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1481439Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1481711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1481808Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1481811Z 2025-12-04T09:43:34.1481910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1482107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1482169Z res = mod(**inputs) 2025-12-04T09:43:34.1482465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1482545Z outputs = self.mobilebert( 2025-12-04T09:43:34.1482847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1482939Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1483216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1483285Z layer_outputs = layer_module( 2025-12-04T09:43:34.1483569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1483728Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1484027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1484131Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1484404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1484507Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1484511Z 2025-12-04T09:43:34.1484609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1484798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1484865Z res = mod(**inputs) 2025-12-04T09:43:34.1485141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1485217Z outputs = self.mobilebert( 2025-12-04T09:43:34.1485499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1485569Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1485857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1485930Z layer_outputs = layer_module( 2025-12-04T09:43:34.1486216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1486299Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1486579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1486655Z self_outputs = self.self( 2025-12-04T09:43:34.1486935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1487006Z self.value(value_tensor) 2025-12-04T09:43:34.1487009Z 2025-12-04T09:43:34.1487117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1487313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1487386Z res = mod(**inputs) 2025-12-04T09:43:34.1487669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1487738Z outputs = self.mobilebert( 2025-12-04T09:43:34.1488025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1488094Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1488381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1488465Z layer_outputs = layer_module( 2025-12-04T09:43:34.1488740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1488904Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1489226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1489335Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1489614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1489692Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1489695Z 2025-12-04T09:43:34.1489799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1489993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1490053Z res = mod(**inputs) 2025-12-04T09:43:34.1490335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1490424Z outputs = self.mobilebert( 2025-12-04T09:43:34.1490708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1490779Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1491053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1491132Z layer_outputs = layer_module( 2025-12-04T09:43:34.1491499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1491678Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1491994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1492110Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1492438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1492528Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1492826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1492931Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1492935Z 2025-12-04T09:43:34.1493043Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1493257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1493326Z res = mod(**inputs) 2025-12-04T09:43:34.1493621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1493706Z outputs = self.mobilebert( 2025-12-04T09:43:34.1494002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1494088Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1494381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1494456Z layer_outputs = layer_module( 2025-12-04T09:43:34.1494761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1494849Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1495165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1495250Z self_outputs = self.self( 2025-12-04T09:43:34.1495551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1495652Z self.query(query_tensor) 2025-12-04T09:43:34.1495656Z 2025-12-04T09:43:34.1495781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1495989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1496065Z res = mod(**inputs) 2025-12-04T09:43:34.1496360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1496435Z outputs = self.mobilebert( 2025-12-04T09:43:34.1496714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1496784Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1497069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1497157Z layer_outputs = layer_module( 2025-12-04T09:43:34.1497437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1497526Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1497804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1497878Z self_outputs = self.self( 2025-12-04T09:43:34.1498154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1498221Z self.key(key_tensor) 2025-12-04T09:43:34.1498226Z 2025-12-04T09:43:34.1498315Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1498394Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1498494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1498696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1498759Z res = mod(**inputs) 2025-12-04T09:43:34.1499041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1499110Z outputs = self.mobilebert( 2025-12-04T09:43:34.1499386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1499463Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1499742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1499818Z layer_outputs = layer_module( 2025-12-04T09:43:34.1500094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1500178Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1500463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1500584Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1500859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1500947Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1500951Z 2025-12-04T09:43:34.1501051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1501267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1501330Z res = mod(**inputs) 2025-12-04T09:43:34.1501607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1501703Z outputs = self.mobilebert( 2025-12-04T09:43:34.1502008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1502088Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1502365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1502435Z layer_outputs = layer_module( 2025-12-04T09:43:34.1502724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1502810Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1503105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1503237Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1503555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1503689Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1503964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1504051Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1504055Z 2025-12-04T09:43:34.1504161Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1504353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1504421Z res = mod(**inputs) 2025-12-04T09:43:34.1504698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1504771Z outputs = self.mobilebert( 2025-12-04T09:43:34.1505054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1505122Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1505398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1505472Z layer_outputs = layer_module( 2025-12-04T09:43:34.1505750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1505849Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1506125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1506231Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1506521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1506602Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1506605Z 2025-12-04T09:43:34.1506712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1506903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1506965Z res = mod(**inputs) 2025-12-04T09:43:34.1507247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1507315Z outputs = self.mobilebert( 2025-12-04T09:43:34.1507618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1507691Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1507974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1508069Z layer_outputs = layer_module( 2025-12-04T09:43:34.1508359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1508452Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1508738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1508846Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1509130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1509240Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1509243Z 2025-12-04T09:43:34.1509343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1509566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1509631Z res = mod(**inputs) 2025-12-04T09:43:34.1509924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1509992Z outputs = self.mobilebert( 2025-12-04T09:43:34.1510263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1510339Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1510610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1510678Z layer_outputs = layer_module( 2025-12-04T09:43:34.1510952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1511045Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1511324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1511445Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1511714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1511804Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1511807Z 2025-12-04T09:43:34.1511904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1512101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1512162Z res = mod(**inputs) 2025-12-04T09:43:34.1512428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1512508Z outputs = self.mobilebert( 2025-12-04T09:43:34.1512776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1512845Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1513122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1513188Z layer_outputs = layer_module( 2025-12-04T09:43:34.1513461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1513577Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1513850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1513995Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1514282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1514409Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1514683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1514771Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1514774Z 2025-12-04T09:43:34.1514880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1515072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1515138Z res = mod(**inputs) 2025-12-04T09:43:34.1515413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1515498Z outputs = self.mobilebert( 2025-12-04T09:43:34.1515774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1515842Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1516109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1516183Z layer_outputs = layer_module( 2025-12-04T09:43:34.1516450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1516546Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1516815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1516921Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1517200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1517280Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1517284Z 2025-12-04T09:43:34.1517389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1517578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1517639Z res = mod(**inputs) 2025-12-04T09:43:34.1517914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1517980Z outputs = self.mobilebert( 2025-12-04T09:43:34.1518251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1518327Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1518599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1518676Z layer_outputs = layer_module( 2025-12-04T09:43:34.1518944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1519032Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1519307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1519411Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1519705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1519813Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1519817Z 2025-12-04T09:43:34.1519933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1520130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1520207Z res = mod(**inputs) 2025-12-04T09:43:34.1520481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1520556Z outputs = self.mobilebert( 2025-12-04T09:43:34.1520826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1520901Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1521173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1521241Z layer_outputs = layer_module( 2025-12-04T09:43:34.1521519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1521627Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1521902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1522020Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1522290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1522373Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1522376Z 2025-12-04T09:43:34.1522475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1522672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1522730Z res = mod(**inputs) 2025-12-04T09:43:34.1522998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1523075Z outputs = self.mobilebert( 2025-12-04T09:43:34.1523345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1523411Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1523685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1523750Z layer_outputs = layer_module( 2025-12-04T09:43:34.1524024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1524112Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1524377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1524504Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1524772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1524894Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1525164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1525250Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1525253Z 2025-12-04T09:43:34.1525356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1525559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1525620Z res = mod(**inputs) 2025-12-04T09:43:34.1525898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1525983Z outputs = self.mobilebert( 2025-12-04T09:43:34.1526277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1526348Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1526621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1526698Z layer_outputs = layer_module( 2025-12-04T09:43:34.1526970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1527068Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1527340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1527447Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1527774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1527854Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1527857Z 2025-12-04T09:43:34.1527959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1528145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1528206Z res = mod(**inputs) 2025-12-04T09:43:34.1528481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1528549Z outputs = self.mobilebert( 2025-12-04T09:43:34.1528818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1528895Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1529165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1529240Z layer_outputs = layer_module( 2025-12-04T09:43:34.1529508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1529594Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1529867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1529970Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1530259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1530366Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1530372Z 2025-12-04T09:43:34.1530472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1530673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1530734Z res = mod(**inputs) 2025-12-04T09:43:34.1531008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1531084Z outputs = self.mobilebert( 2025-12-04T09:43:34.1531426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1531506Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1531808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1531880Z layer_outputs = layer_module( 2025-12-04T09:43:34.1532177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1532309Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1532639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1532775Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1533080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1533180Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1533184Z 2025-12-04T09:43:34.1533296Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1533514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1533587Z res = mod(**inputs) 2025-12-04T09:43:34.1533881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1533988Z outputs = self.mobilebert( 2025-12-04T09:43:34.1534292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1534363Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1534645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1534715Z layer_outputs = layer_module( 2025-12-04T09:43:34.1535001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1535091Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1535369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1535501Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1535781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1535908Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1536185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1536273Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1536277Z 2025-12-04T09:43:34.1536384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1536579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1536642Z res = mod(**inputs) 2025-12-04T09:43:34.1536924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1536996Z outputs = self.mobilebert( 2025-12-04T09:43:34.1537280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1537350Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1537626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1537702Z layer_outputs = layer_module( 2025-12-04T09:43:34.1537980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1538121Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1538398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1538498Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1538501Z 2025-12-04T09:43:34.1538608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1538816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1538880Z res = mod(**inputs) 2025-12-04T09:43:34.1539162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1539230Z outputs = self.mobilebert( 2025-12-04T09:43:34.1539509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1539580Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1539861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1539956Z layer_outputs = layer_module( 2025-12-04T09:43:34.1540233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1540355Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1540631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1540739Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1540743Z 2025-12-04T09:43:34.1540852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1541047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1541109Z res = mod(**inputs) 2025-12-04T09:43:34.1541388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1541461Z outputs = self.mobilebert( 2025-12-04T09:43:34.1541742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1541812Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1542085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1542162Z layer_outputs = layer_module( 2025-12-04T09:43:34.1542436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1542601Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1542875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1542967Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1542973Z 2025-12-04T09:43:34.1543083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1543278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1543348Z res = mod(**inputs) 2025-12-04T09:43:34.1543622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1543691Z outputs = self.mobilebert( 2025-12-04T09:43:34.1543973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1544041Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1544336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1544415Z layer_outputs = layer_module( 2025-12-04T09:43:34.1544714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1544894Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1545178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1545291Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1545557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1545644Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1545649Z 2025-12-04T09:43:34.1545752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1545936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1546016Z res = mod(**inputs) 2025-12-04T09:43:34.1546284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1546351Z outputs = self.mobilebert( 2025-12-04T09:43:34.1546615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1546687Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1546951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1547023Z layer_outputs = layer_module( 2025-12-04T09:43:34.1547287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1547430Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1547701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1547817Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1548088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1548163Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1548167Z 2025-12-04T09:43:34.1548263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1548451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1548510Z res = mod(**inputs) 2025-12-04T09:43:34.1548779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1548843Z outputs = self.mobilebert( 2025-12-04T09:43:34.1549106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1549182Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1549443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1549510Z layer_outputs = layer_module( 2025-12-04T09:43:34.1549778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1549923Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1550219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1550337Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1550623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1550764Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1551034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1551128Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1551132Z 2025-12-04T09:43:34.1551229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1551418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1551487Z res = mod(**inputs) 2025-12-04T09:43:34.1551768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1551836Z outputs = self.mobilebert( 2025-12-04T09:43:34.1552120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1552204Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1552498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1552567Z layer_outputs = layer_module( 2025-12-04T09:43:34.1552849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1553016Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1553302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1553415Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1553700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1553783Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1553789Z 2025-12-04T09:43:34.1553897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1554094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1554163Z res = mod(**inputs) 2025-12-04T09:43:34.1554443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1554511Z outputs = self.mobilebert( 2025-12-04T09:43:34.1554802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1554871Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1555156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1555236Z layer_outputs = layer_module( 2025-12-04T09:43:34.1555526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1555617Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1555892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1555961Z self_outputs = self.self( 2025-12-04T09:43:34.1556251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1556334Z self.value(value_tensor) 2025-12-04T09:43:34.1556338Z 2025-12-04T09:43:34.1556448Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1556640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1556720Z res = mod(**inputs) 2025-12-04T09:43:34.1557034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1557106Z outputs = self.mobilebert( 2025-12-04T09:43:34.1557385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1557463Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1557754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1557831Z layer_outputs = layer_module( 2025-12-04T09:43:34.1558104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1558260Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1558559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1558666Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1558944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1559021Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1559025Z 2025-12-04T09:43:34.1559121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1559317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1559377Z res = mod(**inputs) 2025-12-04T09:43:34.1559646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1559721Z outputs = self.mobilebert( 2025-12-04T09:43:34.1559991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1560065Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1560334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1560400Z layer_outputs = layer_module( 2025-12-04T09:43:34.1560677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1560826Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1561106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1561207Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1561480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1561570Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1561841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1561932Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1561935Z 2025-12-04T09:43:34.1562031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1562220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1562305Z res = mod(**inputs) 2025-12-04T09:43:34.1562577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1562644Z outputs = self.mobilebert( 2025-12-04T09:43:34.1562938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1563021Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1563294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1563361Z layer_outputs = layer_module( 2025-12-04T09:43:34.1563627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1563713Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1563979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1564056Z self_outputs = self.self( 2025-12-04T09:43:34.1564321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1564406Z self.query(query_tensor) 2025-12-04T09:43:34.1564410Z 2025-12-04T09:43:34.1564519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1564707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1564767Z res = mod(**inputs) 2025-12-04T09:43:34.1565044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1565111Z outputs = self.mobilebert( 2025-12-04T09:43:34.1565389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1565460Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1565731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1565810Z layer_outputs = layer_module( 2025-12-04T09:43:34.1566080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1566159Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1566439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1566507Z self_outputs = self.self( 2025-12-04T09:43:34.1566787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1566852Z self.key(key_tensor) 2025-12-04T09:43:34.1566856Z 2025-12-04T09:43:34.1566935Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1567020Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1567118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1567322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1567383Z res = mod(**inputs) 2025-12-04T09:43:34.1567662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1567739Z outputs = self.mobilebert( 2025-12-04T09:43:34.1568015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1568085Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1568387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1568457Z layer_outputs = layer_module( 2025-12-04T09:43:34.1568739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1568840Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1569132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1569262Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1569538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1569625Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1569628Z 2025-12-04T09:43:34.1569728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1569917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1569986Z res = mod(**inputs) 2025-12-04T09:43:34.1570258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1570344Z outputs = self.mobilebert( 2025-12-04T09:43:34.1570633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1570705Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1571000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1571072Z layer_outputs = layer_module( 2025-12-04T09:43:34.1571538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1571637Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1571932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1572067Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1572481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1572614Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1572901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1573002Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1573006Z 2025-12-04T09:43:34.1573112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1573300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1573364Z res = mod(**inputs) 2025-12-04T09:43:34.1573646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1573718Z outputs = self.mobilebert( 2025-12-04T09:43:34.1573995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1574075Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1574349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1574427Z layer_outputs = layer_module( 2025-12-04T09:43:34.1574705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1574797Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1575124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1575236Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1575546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1575655Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1575659Z 2025-12-04T09:43:34.1575761Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1575974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1576036Z res = mod(**inputs) 2025-12-04T09:43:34.1576310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1576388Z outputs = self.mobilebert( 2025-12-04T09:43:34.1576665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1576743Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1577015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1577110Z layer_outputs = layer_module( 2025-12-04T09:43:34.1577394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1577483Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1577760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1577867Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1578136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1578249Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1578253Z 2025-12-04T09:43:34.1578350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1578540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1578607Z res = mod(**inputs) 2025-12-04T09:43:34.1578878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1578951Z outputs = self.mobilebert( 2025-12-04T09:43:34.1579220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1579286Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1579563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1579629Z layer_outputs = layer_module( 2025-12-04T09:43:34.1579906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1579998Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1580269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1580398Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1580665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1580743Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1580754Z 2025-12-04T09:43:34.1580853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1581063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1581133Z res = mod(**inputs) 2025-12-04T09:43:34.1581406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1581499Z outputs = self.mobilebert( 2025-12-04T09:43:34.1581795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1581864Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1582144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1582210Z layer_outputs = layer_module( 2025-12-04T09:43:34.1582480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1582578Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1582850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1582969Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1583266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1583383Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1583661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1583747Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1583751Z 2025-12-04T09:43:34.1583847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1584046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1584105Z res = mod(**inputs) 2025-12-04T09:43:34.1584444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1584516Z outputs = self.mobilebert( 2025-12-04T09:43:34.1584787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1584862Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1585129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1585201Z layer_outputs = layer_module( 2025-12-04T09:43:34.1585471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1585557Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1585832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1585935Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1586207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1586295Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1586298Z 2025-12-04T09:43:34.1586391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1586581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1586640Z res = mod(**inputs) 2025-12-04T09:43:34.1586907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1586981Z outputs = self.mobilebert( 2025-12-04T09:43:34.1587266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1587343Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1587638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1587723Z layer_outputs = layer_module( 2025-12-04T09:43:34.1588004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1588092Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1588367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1588478Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1588751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1588866Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1588869Z 2025-12-04T09:43:34.1588968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1589176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1589246Z res = mod(**inputs) 2025-12-04T09:43:34.1589519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1589594Z outputs = self.mobilebert( 2025-12-04T09:43:34.1589862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1589929Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1590209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1590277Z layer_outputs = layer_module( 2025-12-04T09:43:34.1590544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1590643Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1590911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1591035Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1591302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1591381Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1591385Z 2025-12-04T09:43:34.1591488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1591677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1591744Z res = mod(**inputs) 2025-12-04T09:43:34.1592013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1592082Z outputs = self.mobilebert( 2025-12-04T09:43:34.1592364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1592431Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1592697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1592762Z layer_outputs = layer_module( 2025-12-04T09:43:34.1593022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1593134Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1593401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1593536Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1593822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1593938Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1594220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1594307Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1594310Z 2025-12-04T09:43:34.1594407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1594603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1594663Z res = mod(**inputs) 2025-12-04T09:43:34.1594941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1595037Z outputs = self.mobilebert( 2025-12-04T09:43:34.1595301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1595375Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1595638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1595705Z layer_outputs = layer_module( 2025-12-04T09:43:34.1595985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1596074Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1596358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1596462Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1596727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1596811Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1596815Z 2025-12-04T09:43:34.1596908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1597100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1597159Z res = mod(**inputs) 2025-12-04T09:43:34.1597419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1597491Z outputs = self.mobilebert( 2025-12-04T09:43:34.1597753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1597819Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1598088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1598155Z layer_outputs = layer_module( 2025-12-04T09:43:34.1598431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1598518Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1598790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1598902Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1599191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1599306Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1599325Z 2025-12-04T09:43:34.1599425Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1599629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1599699Z res = mod(**inputs) 2025-12-04T09:43:34.1599970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1600046Z outputs = self.mobilebert( 2025-12-04T09:43:34.1600316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1600381Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1600666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1600732Z layer_outputs = layer_module( 2025-12-04T09:43:34.1601002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1601116Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1601387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1601513Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1601781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1601860Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1601863Z 2025-12-04T09:43:34.1601968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1602155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1602221Z res = mod(**inputs) 2025-12-04T09:43:34.1602490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1602558Z outputs = self.mobilebert( 2025-12-04T09:43:34.1602833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1602899Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1603171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1603245Z layer_outputs = layer_module( 2025-12-04T09:43:34.1603525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1603628Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1603901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1604022Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1604301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1604419Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1604698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1604784Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1604788Z 2025-12-04T09:43:34.1604888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1605114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1605178Z res = mod(**inputs) 2025-12-04T09:43:34.1605459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1605547Z outputs = self.mobilebert( 2025-12-04T09:43:34.1605840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1605921Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1606195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1606265Z layer_outputs = layer_module( 2025-12-04T09:43:34.1606549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1606668Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1606952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1607052Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1607055Z 2025-12-04T09:43:34.1607158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1607359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1607422Z res = mod(**inputs) 2025-12-04T09:43:34.1607708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1607777Z outputs = self.mobilebert( 2025-12-04T09:43:34.1608052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1608131Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1608408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1608477Z layer_outputs = layer_module( 2025-12-04T09:43:34.1608764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1608883Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1609165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1609273Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1609276Z 2025-12-04T09:43:34.1609377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1609578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1609641Z res = mod(**inputs) 2025-12-04T09:43:34.1609928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1610000Z outputs = self.mobilebert( 2025-12-04T09:43:34.1610277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1610353Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1610627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1610694Z layer_outputs = layer_module( 2025-12-04T09:43:34.1610977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1611133Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1611498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1611598Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1611620Z 2025-12-04T09:43:34.1611723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1611998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1612068Z res = mod(**inputs) 2025-12-04T09:43:34.1612378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1612451Z outputs = self.mobilebert( 2025-12-04T09:43:34.1612750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1612833Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1613132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1613205Z layer_outputs = layer_module( 2025-12-04T09:43:34.1613533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1613720Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1614033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1614160Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1614452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1614555Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1614559Z 2025-12-04T09:43:34.1614667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1614880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1614948Z res = mod(**inputs) 2025-12-04T09:43:34.1615245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1615335Z outputs = self.mobilebert( 2025-12-04T09:43:34.1615613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1615689Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1615966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1616034Z layer_outputs = layer_module( 2025-12-04T09:43:34.1616317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1616470Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1616747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1616878Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1617153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1617240Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1617244Z 2025-12-04T09:43:34.1617344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1617539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1617606Z res = mod(**inputs) 2025-12-04T09:43:34.1617896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1617972Z outputs = self.mobilebert( 2025-12-04T09:43:34.1618262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1618332Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1618631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1618700Z layer_outputs = layer_module( 2025-12-04T09:43:34.1618972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1619130Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1619405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1619529Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1619801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1619941Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1620224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1620315Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1620319Z 2025-12-04T09:43:34.1620426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1620620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1620683Z res = mod(**inputs) 2025-12-04T09:43:34.1620975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1621044Z outputs = self.mobilebert( 2025-12-04T09:43:34.1621332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1621405Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1621683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1621761Z layer_outputs = layer_module( 2025-12-04T09:43:34.1622040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1622200Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1622490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1622600Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1622894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1622977Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1622983Z 2025-12-04T09:43:34.1623084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1623287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1623349Z res = mod(**inputs) 2025-12-04T09:43:34.1623637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1623707Z outputs = self.mobilebert( 2025-12-04T09:43:34.1624009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1624091Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1624371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1624465Z layer_outputs = layer_module( 2025-12-04T09:43:34.1624784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1624877Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1625181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1625255Z self_outputs = self.self( 2025-12-04T09:43:34.1625553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1625632Z self.value(value_tensor) 2025-12-04T09:43:34.1625636Z 2025-12-04T09:43:34.1625737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1625937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1626030Z res = mod(**inputs) 2025-12-04T09:43:34.1626306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1626383Z outputs = self.mobilebert( 2025-12-04T09:43:34.1626660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1626730Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1627012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1627080Z layer_outputs = layer_module( 2025-12-04T09:43:34.1627365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1627521Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1627802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1627917Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1628190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1628275Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1628278Z 2025-12-04T09:43:34.1628380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1628578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1628652Z res = mod(**inputs) 2025-12-04T09:43:34.1628943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1629026Z outputs = self.mobilebert( 2025-12-04T09:43:34.1629315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1629390Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1629688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1629761Z layer_outputs = layer_module( 2025-12-04T09:43:34.1630052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1630227Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1630539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1630662Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1630977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1631083Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1631384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1631480Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1631484Z 2025-12-04T09:43:34.1631598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1631805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1631873Z res = mod(**inputs) 2025-12-04T09:43:34.1632175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1632247Z outputs = self.mobilebert( 2025-12-04T09:43:34.1632571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1632647Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1632940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1633020Z layer_outputs = layer_module( 2025-12-04T09:43:34.1633314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1633401Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1633708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1633780Z self_outputs = self.self( 2025-12-04T09:43:34.1634076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1634152Z self.query(query_tensor) 2025-12-04T09:43:34.1634155Z 2025-12-04T09:43:34.1634263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1634474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1634538Z res = mod(**inputs) 2025-12-04T09:43:34.1634831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1634910Z outputs = self.mobilebert( 2025-12-04T09:43:34.1635200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1635277Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1635564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1635641Z layer_outputs = layer_module( 2025-12-04T09:43:34.1635940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1636028Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1636325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1636396Z self_outputs = self.self( 2025-12-04T09:43:34.1636686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1636762Z self.key(key_tensor) 2025-12-04T09:43:34.1636784Z 2025-12-04T09:43:34.1636871Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1636954Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1637070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1637296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1637372Z res = mod(**inputs) 2025-12-04T09:43:34.1637686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1637759Z outputs = self.mobilebert( 2025-12-04T09:43:34.1638066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1638139Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1638446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1638518Z layer_outputs = layer_module( 2025-12-04T09:43:34.1638813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1638923Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1639215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1639344Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1639647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1639733Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1639736Z 2025-12-04T09:43:34.1639849Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1640056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1640124Z res = mod(**inputs) 2025-12-04T09:43:34.1640422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1640499Z outputs = self.mobilebert( 2025-12-04T09:43:34.1640801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1640876Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1641172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1641253Z layer_outputs = layer_module( 2025-12-04T09:43:34.1641550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1641635Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1641939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1642065Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1642372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1642504Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1642799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1642902Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1642905Z 2025-12-04T09:43:34.1643012Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1643222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1643360Z res = mod(**inputs) 2025-12-04T09:43:34.1643653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1643751Z outputs = self.mobilebert( 2025-12-04T09:43:34.1644048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1644139Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1644442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1644516Z layer_outputs = layer_module( 2025-12-04T09:43:34.1644868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1644964Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1645257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1645380Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1645673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1645789Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1645792Z 2025-12-04T09:43:34.1645899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1646108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1646183Z res = mod(**inputs) 2025-12-04T09:43:34.1646485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1646567Z outputs = self.mobilebert( 2025-12-04T09:43:34.1646871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1646945Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1647254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1647331Z layer_outputs = layer_module( 2025-12-04T09:43:34.1647633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1647738Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1648038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1648161Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1648463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1648581Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1648584Z 2025-12-04T09:43:34.1648698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1648910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1648981Z res = mod(**inputs) 2025-12-04T09:43:34.1649282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1649355Z outputs = self.mobilebert( 2025-12-04T09:43:34.1649663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1649735Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1650053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1650134Z layer_outputs = layer_module( 2025-12-04T09:43:34.1650425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1650549Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1650874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1651005Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1651415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1651514Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1651518Z 2025-12-04T09:43:34.1651636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1651850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1651918Z res = mod(**inputs) 2025-12-04T09:43:34.1652225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1652322Z outputs = self.mobilebert( 2025-12-04T09:43:34.1652640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1652724Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1653046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1653146Z layer_outputs = layer_module( 2025-12-04T09:43:34.1653454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1653552Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1653869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1654002Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1654320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1654450Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1654758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1654864Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1654867Z 2025-12-04T09:43:34.1654976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1655195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1655264Z res = mod(**inputs) 2025-12-04T09:43:34.1655570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1655652Z outputs = self.mobilebert( 2025-12-04T09:43:34.1655963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1656037Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1656351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1656424Z layer_outputs = layer_module( 2025-12-04T09:43:34.1656738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1656834Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1657159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1657284Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1657595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1657707Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1657711Z 2025-12-04T09:43:34.1657819Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1658023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1658096Z res = mod(**inputs) 2025-12-04T09:43:34.1658387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1658461Z outputs = self.mobilebert( 2025-12-04T09:43:34.1658766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1658842Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1659149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1659247Z layer_outputs = layer_module( 2025-12-04T09:43:34.1659542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1659645Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1659934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1660053Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1660341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1660456Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1660460Z 2025-12-04T09:43:34.1660580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1660785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1660852Z res = mod(**inputs) 2025-12-04T09:43:34.1661148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1661216Z outputs = self.mobilebert( 2025-12-04T09:43:34.1661500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1661571Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1661846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1661923Z layer_outputs = layer_module( 2025-12-04T09:43:34.1662205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1662311Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1662602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1662732Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1663034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1663118Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1663122Z 2025-12-04T09:43:34.1663232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1663454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1663523Z res = mod(**inputs) 2025-12-04T09:43:34.1663824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1663917Z outputs = self.mobilebert( 2025-12-04T09:43:34.1664233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1664317Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1664609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1664691Z layer_outputs = layer_module( 2025-12-04T09:43:34.1664985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1665080Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1665380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1665527Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1665837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1665957Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1666234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1666332Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1666336Z 2025-12-04T09:43:34.1666433Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1666629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1666699Z res = mod(**inputs) 2025-12-04T09:43:34.1666976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1667054Z outputs = self.mobilebert( 2025-12-04T09:43:34.1667333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1667402Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1667697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1667769Z layer_outputs = layer_module( 2025-12-04T09:43:34.1668065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1668162Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1668455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1668575Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1668874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1671656Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1671667Z 2025-12-04T09:43:34.1671792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1671994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1672057Z res = mod(**inputs) 2025-12-04T09:43:34.1672559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1672636Z outputs = self.mobilebert( 2025-12-04T09:43:34.1672924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1672998Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1673324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1673431Z layer_outputs = layer_module( 2025-12-04T09:43:34.1673751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1673845Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1674131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1674239Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1674516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1674633Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1674639Z 2025-12-04T09:43:34.1674770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1674977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1675042Z res = mod(**inputs) 2025-12-04T09:43:34.1675320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1675399Z outputs = self.mobilebert( 2025-12-04T09:43:34.1675676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1675753Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1676031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1676099Z layer_outputs = layer_module( 2025-12-04T09:43:34.1676386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1676480Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1676759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1676890Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1677167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1677257Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1677261Z 2025-12-04T09:43:34.1677362Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1677554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1677624Z res = mod(**inputs) 2025-12-04T09:43:34.1677904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1677982Z outputs = self.mobilebert( 2025-12-04T09:43:34.1678355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1678428Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1678714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1678783Z layer_outputs = layer_module( 2025-12-04T09:43:34.1679061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1679157Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1679435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1679600Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1679901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1680023Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1680303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1680393Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1680396Z 2025-12-04T09:43:34.1680504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1680700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1680763Z res = mod(**inputs) 2025-12-04T09:43:34.1681048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1681139Z outputs = self.mobilebert( 2025-12-04T09:43:34.1681430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1681500Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1681778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1681854Z layer_outputs = layer_module( 2025-12-04T09:43:34.1682132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1682251Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1682536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1682618Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1682621Z 2025-12-04T09:43:34.1682723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1682921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1682981Z res = mod(**inputs) 2025-12-04T09:43:34.1683265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1683331Z outputs = self.mobilebert( 2025-12-04T09:43:34.1683614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1683684Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1683962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1684040Z layer_outputs = layer_module( 2025-12-04T09:43:34.1684319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1684476Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1684763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1684872Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1684876Z 2025-12-04T09:43:34.1684981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1685177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1685240Z res = mod(**inputs) 2025-12-04T09:43:34.1685530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1685617Z outputs = self.mobilebert( 2025-12-04T09:43:34.1685907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1685992Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1686273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1686357Z layer_outputs = layer_module( 2025-12-04T09:43:34.1686651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1686815Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1687118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1687214Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1687235Z 2025-12-04T09:43:34.1687351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1687560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1687628Z res = mod(**inputs) 2025-12-04T09:43:34.1687931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1688006Z outputs = self.mobilebert( 2025-12-04T09:43:34.1688308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1688382Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1688679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1688760Z layer_outputs = layer_module( 2025-12-04T09:43:34.1689056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1689223Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1689528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1689655Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1689959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1690054Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1690058Z 2025-12-04T09:43:34.1690162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1690374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1690440Z res = mod(**inputs) 2025-12-04T09:43:34.1690742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1690817Z outputs = self.mobilebert( 2025-12-04T09:43:34.1691132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1691280Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1691585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1691662Z layer_outputs = layer_module( 2025-12-04T09:43:34.1691972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1692136Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1692471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1692622Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1692929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1693030Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1693035Z 2025-12-04T09:43:34.1693144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1693364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1693433Z res = mod(**inputs) 2025-12-04T09:43:34.1693742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1693827Z outputs = self.mobilebert( 2025-12-04T09:43:34.1694132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1694238Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1694538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1694613Z layer_outputs = layer_module( 2025-12-04T09:43:34.1694915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1695075Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1695376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1695500Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1695791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1695928Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1696225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1696322Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1696334Z 2025-12-04T09:43:34.1696444Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1696653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1696729Z res = mod(**inputs) 2025-12-04T09:43:34.1697038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1697111Z outputs = self.mobilebert( 2025-12-04T09:43:34.1697412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1697486Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1697804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1697878Z layer_outputs = layer_module( 2025-12-04T09:43:34.1698168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1698345Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1698642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1698756Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1699057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1699163Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1699167Z 2025-12-04T09:43:34.1699300Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1699504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1699567Z res = mod(**inputs) 2025-12-04T09:43:34.1699866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1699938Z outputs = self.mobilebert( 2025-12-04T09:43:34.1700234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1700306Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1700598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1700698Z layer_outputs = layer_module( 2025-12-04T09:43:34.1700996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1701089Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1701376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1701448Z self_outputs = self.self( 2025-12-04T09:43:34.1701747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1701821Z self.value(value_tensor) 2025-12-04T09:43:34.1701824Z 2025-12-04T09:43:34.1701929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1702138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1702206Z res = mod(**inputs) 2025-12-04T09:43:34.1702506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1702579Z outputs = self.mobilebert( 2025-12-04T09:43:34.1702870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1702950Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1703238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1703310Z layer_outputs = layer_module( 2025-12-04T09:43:34.1703610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1703774Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1704080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1704221Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1704515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1704607Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1704611Z 2025-12-04T09:43:34.1704716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1704929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1704995Z res = mod(**inputs) 2025-12-04T09:43:34.1705286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1705389Z outputs = self.mobilebert( 2025-12-04T09:43:34.1705694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1705792Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1706088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1706164Z layer_outputs = layer_module( 2025-12-04T09:43:34.1706464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1706627Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1706922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1707043Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1707359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1707457Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1707751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1707847Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1707850Z 2025-12-04T09:43:34.1707965Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1708166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1708240Z res = mod(**inputs) 2025-12-04T09:43:34.1708533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1708604Z outputs = self.mobilebert( 2025-12-04T09:43:34.1708904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1708979Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1709269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1709349Z layer_outputs = layer_module( 2025-12-04T09:43:34.1709638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1709731Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1710019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1710090Z self_outputs = self.self( 2025-12-04T09:43:34.1710387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1710462Z self.query(query_tensor) 2025-12-04T09:43:34.1710466Z 2025-12-04T09:43:34.1710579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1710804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1710873Z res = mod(**inputs) 2025-12-04T09:43:34.1711178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1711250Z outputs = self.mobilebert( 2025-12-04T09:43:34.1711549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1711628Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1711928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1712029Z layer_outputs = layer_module( 2025-12-04T09:43:34.1712335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1712428Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1712728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1712800Z self_outputs = self.self( 2025-12-04T09:43:34.1713100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1713170Z self.key(key_tensor) 2025-12-04T09:43:34.1713174Z 2025-12-04T09:43:34.1713258Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1713349Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1713456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1713677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1713753Z res = mod(**inputs) 2025-12-04T09:43:34.1714046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1714127Z outputs = self.mobilebert( 2025-12-04T09:43:34.1714416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1714489Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1714787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1714861Z layer_outputs = layer_module( 2025-12-04T09:43:34.1715158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1715246Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1715538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1715670Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1715962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1716048Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1716060Z 2025-12-04T09:43:34.1716164Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1716365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1716435Z res = mod(**inputs) 2025-12-04T09:43:34.1716727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1716801Z outputs = self.mobilebert( 2025-12-04T09:43:34.1717097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1717190Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1717490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1717562Z layer_outputs = layer_module( 2025-12-04T09:43:34.1717854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1717945Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1718239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1718364Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1718685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1718836Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1719140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1719236Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1719239Z 2025-12-04T09:43:34.1719345Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1719558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1719625Z res = mod(**inputs) 2025-12-04T09:43:34.1719932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1720005Z outputs = self.mobilebert( 2025-12-04T09:43:34.1720316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1720399Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1720695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1720768Z layer_outputs = layer_module( 2025-12-04T09:43:34.1721064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1721161Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1721460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1721574Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1721869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1721967Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1721971Z 2025-12-04T09:43:34.1722077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1722290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1722355Z res = mod(**inputs) 2025-12-04T09:43:34.1722647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1722729Z outputs = self.mobilebert( 2025-12-04T09:43:34.1723021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1723102Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1723397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1723472Z layer_outputs = layer_module( 2025-12-04T09:43:34.1723795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1723895Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1724188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1724309Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1724600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1724721Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1724725Z 2025-12-04T09:43:34.1724830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1725069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1725142Z res = mod(**inputs) 2025-12-04T09:43:34.1725450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1725531Z outputs = self.mobilebert( 2025-12-04T09:43:34.1725827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1725900Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1726200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1726272Z layer_outputs = layer_module( 2025-12-04T09:43:34.1726563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1726694Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1726994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1727134Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1727430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1727516Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1727519Z 2025-12-04T09:43:34.1727631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1727837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1727909Z res = mod(**inputs) 2025-12-04T09:43:34.1728202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1728277Z outputs = self.mobilebert( 2025-12-04T09:43:34.1728582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1728655Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1728951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1729031Z layer_outputs = layer_module( 2025-12-04T09:43:34.1729328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1729430Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1729726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1729859Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1730164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1730312Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1730623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1730719Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1730722Z 2025-12-04T09:43:34.1730836Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1731055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1731123Z res = mod(**inputs) 2025-12-04T09:43:34.1731528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1731666Z outputs = self.mobilebert( 2025-12-04T09:43:34.1732009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1732101Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1732413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1732506Z layer_outputs = layer_module( 2025-12-04T09:43:34.1732800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1732897Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1733202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1733319Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1733665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1733756Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1733760Z 2025-12-04T09:43:34.1733872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1734087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1734154Z res = mod(**inputs) 2025-12-04T09:43:34.1734463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1734546Z outputs = self.mobilebert( 2025-12-04T09:43:34.1734858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1734943Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1735246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1735324Z layer_outputs = layer_module( 2025-12-04T09:43:34.1735639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1735738Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1736043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1736160Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1736456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1736580Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1736584Z 2025-12-04T09:43:34.1736694Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1736905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1736982Z res = mod(**inputs) 2025-12-04T09:43:34.1737303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1737387Z outputs = self.mobilebert( 2025-12-04T09:43:34.1737691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1737767Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1738076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1738151Z layer_outputs = layer_module( 2025-12-04T09:43:34.1738461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1738581Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1738900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1739042Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1739343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1739430Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1739442Z 2025-12-04T09:43:34.1739551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1739762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1739834Z res = mod(**inputs) 2025-12-04T09:43:34.1740142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1740238Z outputs = self.mobilebert( 2025-12-04T09:43:34.1740552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1740629Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1740936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1741011Z layer_outputs = layer_module( 2025-12-04T09:43:34.1741312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1741416Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1741726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1741867Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1742175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1742305Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1742614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1742712Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1742716Z 2025-12-04T09:43:34.1742824Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1743042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1743109Z res = mod(**inputs) 2025-12-04T09:43:34.1743419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1743495Z outputs = self.mobilebert( 2025-12-04T09:43:34.1743796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1743908Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1744198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1744274Z layer_outputs = layer_module( 2025-12-04T09:43:34.1744551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1744647Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1744947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1745062Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1745376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1745491Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1745496Z 2025-12-04T09:43:34.1745604Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1745819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1745896Z res = mod(**inputs) 2025-12-04T09:43:34.1746171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1746248Z outputs = self.mobilebert( 2025-12-04T09:43:34.1746524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1746600Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1746899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1746968Z layer_outputs = layer_module( 2025-12-04T09:43:34.1747260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1747351Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1747627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1747740Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1748017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1748132Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1748137Z 2025-12-04T09:43:34.1748239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1748436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1748509Z res = mod(**inputs) 2025-12-04T09:43:34.1748791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1748868Z outputs = self.mobilebert( 2025-12-04T09:43:34.1749145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1749214Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1749501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1749569Z layer_outputs = layer_module( 2025-12-04T09:43:34.1749850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1749949Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1750316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1750449Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1750728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1750811Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1750814Z 2025-12-04T09:43:34.1750921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1751116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1751184Z res = mod(**inputs) 2025-12-04T09:43:34.1751467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1751559Z outputs = self.mobilebert( 2025-12-04T09:43:34.1751872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1751945Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1752234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1752303Z layer_outputs = layer_module( 2025-12-04T09:43:34.1752599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1752697Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1752975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1753118Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1753405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1753524Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1753803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1753891Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1753894Z 2025-12-04T09:43:34.1753994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1754193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1754254Z res = mod(**inputs) 2025-12-04T09:43:34.1754535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1754605Z outputs = self.mobilebert( 2025-12-04T09:43:34.1754888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1754963Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1755241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1755307Z layer_outputs = layer_module( 2025-12-04T09:43:34.1755589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1755707Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1755987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1756070Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1756074Z 2025-12-04T09:43:34.1756174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1756390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1756455Z res = mod(**inputs) 2025-12-04T09:43:34.1756747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1756815Z outputs = self.mobilebert( 2025-12-04T09:43:34.1757098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1757174Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1757469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1757556Z layer_outputs = layer_module( 2025-12-04T09:43:34.1757839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1757971Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1758253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1758362Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1758366Z 2025-12-04T09:43:34.1758464Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1758662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1758722Z res = mod(**inputs) 2025-12-04T09:43:34.1759003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1759072Z outputs = self.mobilebert( 2025-12-04T09:43:34.1759374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1759454Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1759730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1759807Z layer_outputs = layer_module( 2025-12-04T09:43:34.1760084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1760239Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1760527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1760615Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1760621Z 2025-12-04T09:43:34.1760719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1760917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1760978Z res = mod(**inputs) 2025-12-04T09:43:34.1761251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1761319Z outputs = self.mobilebert( 2025-12-04T09:43:34.1761592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1761666Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1761937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1762010Z layer_outputs = layer_module( 2025-12-04T09:43:34.1762281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1762434Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1762729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1762846Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1763113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1763207Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1763210Z 2025-12-04T09:43:34.1763306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1763498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1763574Z res = mod(**inputs) 2025-12-04T09:43:34.1763844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1763933Z outputs = self.mobilebert( 2025-12-04T09:43:34.1764204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1764279Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1764549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1764615Z layer_outputs = layer_module( 2025-12-04T09:43:34.1764893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1765042Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1765320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1765459Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1765726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1765812Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1765816Z 2025-12-04T09:43:34.1765913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1766099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1766166Z res = mod(**inputs) 2025-12-04T09:43:34.1766430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1766505Z outputs = self.mobilebert( 2025-12-04T09:43:34.1766777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1766844Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1767122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1767189Z layer_outputs = layer_module( 2025-12-04T09:43:34.1767462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1767607Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1767873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1767994Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1768259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1768383Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1768666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1768756Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1768759Z 2025-12-04T09:43:34.1768866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1769055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1769115Z res = mod(**inputs) 2025-12-04T09:43:34.1769391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1769457Z outputs = self.mobilebert( 2025-12-04T09:43:34.1769730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1769816Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1770101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1770177Z layer_outputs = layer_module( 2025-12-04T09:43:34.1770445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1770606Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1770878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1770982Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1771325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1771437Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1771441Z 2025-12-04T09:43:34.1771550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1771745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1771808Z res = mod(**inputs) 2025-12-04T09:43:34.1772091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1772159Z outputs = self.mobilebert( 2025-12-04T09:43:34.1772589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1772679Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1772984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1773072Z layer_outputs = layer_module( 2025-12-04T09:43:34.1773376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1773469Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1773776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1773851Z self_outputs = self.self( 2025-12-04T09:43:34.1774151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1774246Z self.value(value_tensor) 2025-12-04T09:43:34.1774251Z 2025-12-04T09:43:34.1774356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1774568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1774638Z res = mod(**inputs) 2025-12-04T09:43:34.1774932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1775061Z outputs = self.mobilebert( 2025-12-04T09:43:34.1775356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1775437Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1775727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1775799Z layer_outputs = layer_module( 2025-12-04T09:43:34.1776097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1776260Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1776599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1776749Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1777043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1777137Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1777141Z 2025-12-04T09:43:34.1777249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1777461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1777536Z res = mod(**inputs) 2025-12-04T09:43:34.1777826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1777908Z outputs = self.mobilebert( 2025-12-04T09:43:34.1778225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1778302Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1778607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1778682Z layer_outputs = layer_module( 2025-12-04T09:43:34.1778982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1779145Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1779436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1779557Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1779852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1779942Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1780242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1780338Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1780342Z 2025-12-04T09:43:34.1780455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1780658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1780723Z res = mod(**inputs) 2025-12-04T09:43:34.1781029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1781101Z outputs = self.mobilebert( 2025-12-04T09:43:34.1781402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1781478Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1782197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1782283Z layer_outputs = layer_module( 2025-12-04T09:43:34.1782578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1782669Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1782971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1783044Z self_outputs = self.self( 2025-12-04T09:43:34.1783343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1783438Z self.query(query_tensor) 2025-12-04T09:43:34.1783443Z 2025-12-04T09:43:34.1783571Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1783788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1783857Z res = mod(**inputs) 2025-12-04T09:43:34.1784164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1784239Z outputs = self.mobilebert( 2025-12-04T09:43:34.1784535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1784618Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1784912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1785012Z layer_outputs = layer_module( 2025-12-04T09:43:34.1785314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1785405Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1785701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1785773Z self_outputs = self.self( 2025-12-04T09:43:34.1786065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1786141Z self.key(key_tensor) 2025-12-04T09:43:34.1786144Z 2025-12-04T09:43:34.1786230Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1786317Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1786422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1786631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1786703Z res = mod(**inputs) 2025-12-04T09:43:34.1786996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1787068Z outputs = self.mobilebert( 2025-12-04T09:43:34.1787366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1787438Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1787737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1787808Z layer_outputs = layer_module( 2025-12-04T09:43:34.1788097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1788192Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1788486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1788631Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1788930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1789017Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1789020Z 2025-12-04T09:43:34.1789132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1789336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1789400Z res = mod(**inputs) 2025-12-04T09:43:34.1789700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1789796Z outputs = self.mobilebert( 2025-12-04T09:43:34.1790111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1790191Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1790483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1790564Z layer_outputs = layer_module( 2025-12-04T09:43:34.1790855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1790939Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1791237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1791365Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1791680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1791814Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1792107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1792213Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1792217Z 2025-12-04T09:43:34.1792322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1792531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1792597Z res = mod(**inputs) 2025-12-04T09:43:34.1792886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1792967Z outputs = self.mobilebert( 2025-12-04T09:43:34.1793258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1793339Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1793631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1793705Z layer_outputs = layer_module( 2025-12-04T09:43:34.1793998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1794095Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1794387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1794510Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1794802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1794895Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1794900Z 2025-12-04T09:43:34.1795020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1795227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1795301Z res = mod(**inputs) 2025-12-04T09:43:34.1795598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1795677Z outputs = self.mobilebert( 2025-12-04T09:43:34.1795964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1796037Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1796333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1796428Z layer_outputs = layer_module( 2025-12-04T09:43:34.1796742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1796849Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1797146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1797268Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1797559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1797677Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1797680Z 2025-12-04T09:43:34.1797798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1798024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1798096Z res = mod(**inputs) 2025-12-04T09:43:34.1798393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1798463Z outputs = self.mobilebert( 2025-12-04T09:43:34.1798756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1798826Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1799111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1799181Z layer_outputs = layer_module( 2025-12-04T09:43:34.1799459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1799561Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1799836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1799962Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1800246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1800327Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1800331Z 2025-12-04T09:43:34.1800435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1800630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1800690Z res = mod(**inputs) 2025-12-04T09:43:34.1800975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1801046Z outputs = self.mobilebert( 2025-12-04T09:43:34.1801331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1801427Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1801704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1801781Z layer_outputs = layer_module( 2025-12-04T09:43:34.1802056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1802146Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1802430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1802552Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1802876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1802998Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1803274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1803373Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1803377Z 2025-12-04T09:43:34.1803477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1803675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1803738Z res = mod(**inputs) 2025-12-04T09:43:34.1804010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1804103Z outputs = self.mobilebert( 2025-12-04T09:43:34.1804379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1804450Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1804735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1804803Z layer_outputs = layer_module( 2025-12-04T09:43:34.1805085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1805174Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1805452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1805568Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1805850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1805938Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1805941Z 2025-12-04T09:43:34.1806044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1806239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1806309Z res = mod(**inputs) 2025-12-04T09:43:34.1806585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1806660Z outputs = self.mobilebert( 2025-12-04T09:43:34.1806936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1807005Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1807289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1807360Z layer_outputs = layer_module( 2025-12-04T09:43:34.1807653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1807753Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1808029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1808143Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1808418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1808527Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1808531Z 2025-12-04T09:43:34.1808658Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1808858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1808942Z res = mod(**inputs) 2025-12-04T09:43:34.1809222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1809291Z outputs = self.mobilebert( 2025-12-04T09:43:34.1809576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1809646Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1809934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1810010Z layer_outputs = layer_module( 2025-12-04T09:43:34.1810289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1810406Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1810699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1810831Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1811135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1811292Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1811299Z 2025-12-04T09:43:34.1811426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1811638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1811708Z res = mod(**inputs) 2025-12-04T09:43:34.1812017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1812098Z outputs = self.mobilebert( 2025-12-04T09:43:34.1812416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1812503Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1812808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1812903Z layer_outputs = layer_module( 2025-12-04T09:43:34.1813196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1813291Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1813598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1813730Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1814101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1814252Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1814550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1814657Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1814660Z 2025-12-04T09:43:34.1814769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1814981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1815048Z res = mod(**inputs) 2025-12-04T09:43:34.1815343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1815444Z outputs = self.mobilebert( 2025-12-04T09:43:34.1815762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1815840Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1816140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1816214Z layer_outputs = layer_module( 2025-12-04T09:43:34.1816571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1816666Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1816963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1817086Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1817399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1817494Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1817499Z 2025-12-04T09:43:34.1817606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1817844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1817917Z res = mod(**inputs) 2025-12-04T09:43:34.1818212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1818284Z outputs = self.mobilebert( 2025-12-04T09:43:34.1818585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1818656Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1818958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1819032Z layer_outputs = layer_module( 2025-12-04T09:43:34.1819324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1819428Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1819719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1819839Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1820130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1820242Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1820247Z 2025-12-04T09:43:34.1820361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1820563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1820630Z res = mod(**inputs) 2025-12-04T09:43:34.1820946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1821020Z outputs = self.mobilebert( 2025-12-04T09:43:34.1821318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1821391Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1821682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1821762Z layer_outputs = layer_module( 2025-12-04T09:43:34.1822058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1822187Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1822497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1822629Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1822931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1823016Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1823019Z 2025-12-04T09:43:34.1823133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1823337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1823404Z res = mod(**inputs) 2025-12-04T09:43:34.1823705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1823807Z outputs = self.mobilebert( 2025-12-04T09:43:34.1824110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1824190Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1824485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1824565Z layer_outputs = layer_module( 2025-12-04T09:43:34.1824864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1824957Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1825263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1825388Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1825680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1825800Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1826079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1826176Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1826179Z 2025-12-04T09:43:34.1826277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1826472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1826539Z res = mod(**inputs) 2025-12-04T09:43:34.1826822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1826902Z outputs = self.mobilebert( 2025-12-04T09:43:34.1827204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1827275Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1827561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1827630Z layer_outputs = layer_module( 2025-12-04T09:43:34.1827914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1828034Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1828310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1828421Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1828425Z 2025-12-04T09:43:34.1828526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1828747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1828813Z res = mod(**inputs) 2025-12-04T09:43:34.1829088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1829165Z outputs = self.mobilebert( 2025-12-04T09:43:34.1829444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1829514Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1829799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1829869Z layer_outputs = layer_module( 2025-12-04T09:43:34.1830171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1830289Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1830572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1830688Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1830692Z 2025-12-04T09:43:34.1830791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1830993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1831056Z res = mod(**inputs) 2025-12-04T09:43:34.1831331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1831410Z outputs = self.mobilebert( 2025-12-04T09:43:34.1831686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1831755Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1832041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1832111Z layer_outputs = layer_module( 2025-12-04T09:43:34.1832394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1832551Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1832826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1832926Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1832933Z 2025-12-04T09:43:34.1833033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1833233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1833339Z res = mod(**inputs) 2025-12-04T09:43:34.1833616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1833695Z outputs = self.mobilebert( 2025-12-04T09:43:34.1833972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1834042Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1834345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1834418Z layer_outputs = layer_module( 2025-12-04T09:43:34.1834721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1834923Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1835221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1835355Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1835644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1835745Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1835749Z 2025-12-04T09:43:34.1835855Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1836058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1836127Z res = mod(**inputs) 2025-12-04T09:43:34.1836420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1836491Z outputs = self.mobilebert( 2025-12-04T09:43:34.1836775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1836844Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1837125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1837192Z layer_outputs = layer_module( 2025-12-04T09:43:34.1837463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1837622Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1837895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1838024Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1838299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1838380Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1838384Z 2025-12-04T09:43:34.1838489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1838679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1838747Z res = mod(**inputs) 2025-12-04T09:43:34.1839019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1839089Z outputs = self.mobilebert( 2025-12-04T09:43:34.1839372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1839443Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1839734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1839810Z layer_outputs = layer_module( 2025-12-04T09:43:34.1840085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1840242Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1840516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1840636Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1840916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1841055Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1841357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1841449Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1841453Z 2025-12-04T09:43:34.1841553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1841754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1841815Z res = mod(**inputs) 2025-12-04T09:43:34.1842102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1842172Z outputs = self.mobilebert( 2025-12-04T09:43:34.1842448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1842543Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1842826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1842897Z layer_outputs = layer_module( 2025-12-04T09:43:34.1843184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1843341Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1843627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1843735Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1844067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1844157Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1844161Z 2025-12-04T09:43:34.1844260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1844455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1844518Z res = mod(**inputs) 2025-12-04T09:43:34.1844787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1844861Z outputs = self.mobilebert( 2025-12-04T09:43:34.1845136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1845203Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1845490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1845563Z layer_outputs = layer_module( 2025-12-04T09:43:34.1845867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1845954Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1846230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1846305Z self_outputs = self.self( 2025-12-04T09:43:34.1846587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1846664Z self.value(value_tensor) 2025-12-04T09:43:34.1846668Z 2025-12-04T09:43:34.1846767Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1846959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1847047Z res = mod(**inputs) 2025-12-04T09:43:34.1847375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1847453Z outputs = self.mobilebert( 2025-12-04T09:43:34.1847755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1847831Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1848129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1848202Z layer_outputs = layer_module( 2025-12-04T09:43:34.1848496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1848671Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1848985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1849109Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1849402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1849486Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1849490Z 2025-12-04T09:43:34.1849605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1849808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1849880Z res = mod(**inputs) 2025-12-04T09:43:34.1850171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1850247Z outputs = self.mobilebert( 2025-12-04T09:43:34.1850549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1850622Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1850914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1850996Z layer_outputs = layer_module( 2025-12-04T09:43:34.1851366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1851549Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1851844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1851958Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1852274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1852366Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1852701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1852801Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1852805Z 2025-12-04T09:43:34.1852915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1853140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1853208Z res = mod(**inputs) 2025-12-04T09:43:34.1853519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1853600Z outputs = self.mobilebert( 2025-12-04T09:43:34.1853898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1853991Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1854269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1854339Z layer_outputs = layer_module( 2025-12-04T09:43:34.1854624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1854708Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1854996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1855065Z self_outputs = self.self( 2025-12-04T09:43:34.1855338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1855435Z self.query(query_tensor) 2025-12-04T09:43:34.1855439Z 2025-12-04T09:43:34.1855540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1855735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1855804Z res = mod(**inputs) 2025-12-04T09:43:34.1856076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1856149Z outputs = self.mobilebert( 2025-12-04T09:43:34.1856422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1856491Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1856775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1856848Z layer_outputs = layer_module( 2025-12-04T09:43:34.1857131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1857214Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1857491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1857569Z self_outputs = self.self( 2025-12-04T09:43:34.1857851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1857917Z self.key(key_tensor) 2025-12-04T09:43:34.1857926Z 2025-12-04T09:43:34.1858004Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1858081Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1858188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1858379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1858440Z res = mod(**inputs) 2025-12-04T09:43:34.1858737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1858807Z outputs = self.mobilebert( 2025-12-04T09:43:34.1859079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1859155Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1859424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1859501Z layer_outputs = layer_module( 2025-12-04T09:43:34.1859769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1859869Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1860163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1860284Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1860560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1860640Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1860643Z 2025-12-04T09:43:34.1860741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1860941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1861001Z res = mod(**inputs) 2025-12-04T09:43:34.1861268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1861360Z outputs = self.mobilebert( 2025-12-04T09:43:34.1861633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1861708Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1861979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1862046Z layer_outputs = layer_module( 2025-12-04T09:43:34.1862324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1862401Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1862679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1862794Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1863066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1863194Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1863475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1863575Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1863579Z 2025-12-04T09:43:34.1863674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1863859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1863925Z res = mod(**inputs) 2025-12-04T09:43:34.1864187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1864251Z outputs = self.mobilebert( 2025-12-04T09:43:34.1864524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1864591Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1864880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1864947Z layer_outputs = layer_module( 2025-12-04T09:43:34.1865212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1865307Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1865569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1865679Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1865939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1866035Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1866054Z 2025-12-04T09:43:34.1866159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1866343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1866403Z res = mod(**inputs) 2025-12-04T09:43:34.1866677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1866743Z outputs = self.mobilebert( 2025-12-04T09:43:34.1867012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1867080Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1867341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1867436Z layer_outputs = layer_module( 2025-12-04T09:43:34.1867703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1867799Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1868058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1868159Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1868426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1868530Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1868533Z 2025-12-04T09:43:34.1868632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1868819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1868877Z res = mod(**inputs) 2025-12-04T09:43:34.1869146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1869211Z outputs = self.mobilebert( 2025-12-04T09:43:34.1869470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1869542Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1869801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1869873Z layer_outputs = layer_module( 2025-12-04T09:43:34.1870133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1870221Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1870489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1870625Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1870894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1870967Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1870971Z 2025-12-04T09:43:34.1871064Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1871249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1871306Z res = mod(**inputs) 2025-12-04T09:43:34.1871564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1871665Z outputs = self.mobilebert( 2025-12-04T09:43:34.1871939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1872014Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1872386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1872462Z layer_outputs = layer_module( 2025-12-04T09:43:34.1872737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1872825Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1873093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1873211Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1873517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1873646Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1873915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1874002Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1874013Z 2025-12-04T09:43:34.1874113Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1874302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1874370Z res = mod(**inputs) 2025-12-04T09:43:34.1874642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1874715Z outputs = self.mobilebert( 2025-12-04T09:43:34.1874996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1875066Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1875350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1875420Z layer_outputs = layer_module( 2025-12-04T09:43:34.1875691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1875792Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1876070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1876179Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1876472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1876552Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1876555Z 2025-12-04T09:43:34.1876683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1876867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1876927Z res = mod(**inputs) 2025-12-04T09:43:34.1877195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1877261Z outputs = self.mobilebert( 2025-12-04T09:43:34.1877533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1877602Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1877871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1877972Z layer_outputs = layer_module( 2025-12-04T09:43:34.1878271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1878368Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1878641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1878747Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1879025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1879133Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1879137Z 2025-12-04T09:43:34.1879235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1879448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1879511Z res = mod(**inputs) 2025-12-04T09:43:34.1879787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1879857Z outputs = self.mobilebert( 2025-12-04T09:43:34.1880124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1880198Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1880466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1880540Z layer_outputs = layer_module( 2025-12-04T09:43:34.1880809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1880899Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1881178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1881297Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1881567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1881651Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1881654Z 2025-12-04T09:43:34.1881750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1881943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1882005Z res = mod(**inputs) 2025-12-04T09:43:34.1882277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1882355Z outputs = self.mobilebert( 2025-12-04T09:43:34.1882656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1882736Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1883012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1883080Z layer_outputs = layer_module( 2025-12-04T09:43:34.1883369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1883458Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1883734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1883876Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1884169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1884297Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1884576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1884665Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1884668Z 2025-12-04T09:43:34.1884773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1884965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1885032Z res = mod(**inputs) 2025-12-04T09:43:34.1885306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1885396Z outputs = self.mobilebert( 2025-12-04T09:43:34.1885684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1885754Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1886041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1886109Z layer_outputs = layer_module( 2025-12-04T09:43:34.1886386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1886482Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1886764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1886872Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1887163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1887245Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1887250Z 2025-12-04T09:43:34.1887352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1887545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1887605Z res = mod(**inputs) 2025-12-04T09:43:34.1887888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1887955Z outputs = self.mobilebert( 2025-12-04T09:43:34.1888238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1888307Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1888588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1888666Z layer_outputs = layer_module( 2025-12-04T09:43:34.1888960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1889052Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1889338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1889445Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1889728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1889836Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1889855Z 2025-12-04T09:43:34.1889959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1890168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1890249Z res = mod(**inputs) 2025-12-04T09:43:34.1890553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1890629Z outputs = self.mobilebert( 2025-12-04T09:43:34.1890919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1891000Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1891348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1891428Z layer_outputs = layer_module( 2025-12-04T09:43:34.1891730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1891854Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1892158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1892288Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1892577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1892671Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1892675Z 2025-12-04T09:43:34.1892781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1893001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1893069Z res = mod(**inputs) 2025-12-04T09:43:34.1893369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1893467Z outputs = self.mobilebert( 2025-12-04T09:43:34.1893759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1893841Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1894132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1894205Z layer_outputs = layer_module( 2025-12-04T09:43:34.1894509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1894602Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1894894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1895035Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1895328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1895493Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1895788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1895881Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1895885Z 2025-12-04T09:43:34.1895994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1896199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1896272Z res = mod(**inputs) 2025-12-04T09:43:34.1896563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1896657Z outputs = self.mobilebert( 2025-12-04T09:43:34.1896973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1897052Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1897346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1897424Z layer_outputs = layer_module( 2025-12-04T09:43:34.1897720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1897854Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1898146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1898249Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1898253Z 2025-12-04T09:43:34.1898366Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1898574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1898649Z res = mod(**inputs) 2025-12-04T09:43:34.1898941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1899014Z outputs = self.mobilebert( 2025-12-04T09:43:34.1899310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1899384Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1899677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1899757Z layer_outputs = layer_module( 2025-12-04T09:43:34.1900051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1900181Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1900473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1900587Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1900590Z 2025-12-04T09:43:34.1900703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1900906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1900976Z res = mod(**inputs) 2025-12-04T09:43:34.1901265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1901339Z outputs = self.mobilebert( 2025-12-04T09:43:34.1901639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1901713Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1902024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1902106Z layer_outputs = layer_module( 2025-12-04T09:43:34.1902399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1902571Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1902861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1902957Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1902988Z 2025-12-04T09:43:34.1903096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1903315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1903389Z res = mod(**inputs) 2025-12-04T09:43:34.1903681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1903752Z outputs = self.mobilebert( 2025-12-04T09:43:34.1904054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1904126Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1904425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1904495Z layer_outputs = layer_module( 2025-12-04T09:43:34.1904788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1904982Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1905279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1905407Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1905717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1905806Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1905810Z 2025-12-04T09:43:34.1905913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1906109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1906171Z res = mod(**inputs) 2025-12-04T09:43:34.1906457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1906528Z outputs = self.mobilebert( 2025-12-04T09:43:34.1906817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1906886Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1907162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1907238Z layer_outputs = layer_module( 2025-12-04T09:43:34.1907514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1907666Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1907954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1908078Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1908382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1908465Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1908469Z 2025-12-04T09:43:34.1908572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1908772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1908835Z res = mod(**inputs) 2025-12-04T09:43:34.1909122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1909191Z outputs = self.mobilebert( 2025-12-04T09:43:34.1909469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1909566Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1909862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1909941Z layer_outputs = layer_module( 2025-12-04T09:43:34.1910216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1910367Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1910650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1910769Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1911044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1911188Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1911467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1911563Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1911566Z 2025-12-04T09:43:34.1911666Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1911857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1911929Z res = mod(**inputs) 2025-12-04T09:43:34.1912203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1912278Z outputs = self.mobilebert( 2025-12-04T09:43:34.1912555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1912625Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1912914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1912982Z layer_outputs = layer_module( 2025-12-04T09:43:34.1913271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1913425Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1913702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1913817Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1914093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1914176Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1914187Z 2025-12-04T09:43:34.1914289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1914497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1914570Z res = mod(**inputs) 2025-12-04T09:43:34.1914852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1914919Z outputs = self.mobilebert( 2025-12-04T09:43:34.1915204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1915272Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1915558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1915648Z layer_outputs = layer_module( 2025-12-04T09:43:34.1915940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1916035Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1916316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1916385Z self_outputs = self.self( 2025-12-04T09:43:34.1916670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1916739Z self.value(value_tensor) 2025-12-04T09:43:34.1916743Z 2025-12-04T09:43:34.1916850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1917044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1917134Z res = mod(**inputs) 2025-12-04T09:43:34.1917419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1917489Z outputs = self.mobilebert( 2025-12-04T09:43:34.1917771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1917839Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1918115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1918192Z layer_outputs = layer_module( 2025-12-04T09:43:34.1918466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1918619Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1918905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1919014Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1919299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1919378Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1919381Z 2025-12-04T09:43:34.1919480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1919682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1919742Z res = mod(**inputs) 2025-12-04T09:43:34.1920024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1920092Z outputs = self.mobilebert( 2025-12-04T09:43:34.1920372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1920449Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1920743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1920813Z layer_outputs = layer_module( 2025-12-04T09:43:34.1921096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1921248Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1921533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1921639Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1921934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1922039Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1922317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1922415Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1922419Z 2025-12-04T09:43:34.1922518Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1922709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1922780Z res = mod(**inputs) 2025-12-04T09:43:34.1923054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1923130Z outputs = self.mobilebert( 2025-12-04T09:43:34.1923420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1923511Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1923817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1923891Z layer_outputs = layer_module( 2025-12-04T09:43:34.1924184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1924279Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1924577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1924655Z self_outputs = self.self( 2025-12-04T09:43:34.1924954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1925028Z self.query(query_tensor) 2025-12-04T09:43:34.1925032Z 2025-12-04T09:43:34.1925147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1925353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1925425Z res = mod(**inputs) 2025-12-04T09:43:34.1925720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1925791Z outputs = self.mobilebert( 2025-12-04T09:43:34.1926091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1926165Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1926460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1926544Z layer_outputs = layer_module( 2025-12-04T09:43:34.1926841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1926953Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1927247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1927321Z self_outputs = self.self( 2025-12-04T09:43:34.1927621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.1927690Z self.key(key_tensor) 2025-12-04T09:43:34.1927694Z 2025-12-04T09:43:34.1927786Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1927868Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.1927974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1928208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1928276Z res = mod(**inputs) 2025-12-04T09:43:34.1928586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1928670Z outputs = self.mobilebert( 2025-12-04T09:43:34.1928970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1929053Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1929355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1929427Z layer_outputs = layer_module( 2025-12-04T09:43:34.1929736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1929872Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1930167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1930303Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1930595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.1930687Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1930691Z 2025-12-04T09:43:34.1930796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1931006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1931082Z res = mod(**inputs) 2025-12-04T09:43:34.1931459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1931557Z outputs = self.mobilebert( 2025-12-04T09:43:34.1931866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1931945Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1932257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1932333Z layer_outputs = layer_module( 2025-12-04T09:43:34.1932642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1932742Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1933034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.1933169Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.1933464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.1933596Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1933923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1934021Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1934026Z 2025-12-04T09:43:34.1934139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1934343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1934407Z res = mod(**inputs) 2025-12-04T09:43:34.1934711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1934783Z outputs = self.mobilebert( 2025-12-04T09:43:34.1935100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1935195Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1935491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1935568Z layer_outputs = layer_module( 2025-12-04T09:43:34.1935857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1935954Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1936257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1936372Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1936670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1936777Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1936782Z 2025-12-04T09:43:34.1936888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1937103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1937169Z res = mod(**inputs) 2025-12-04T09:43:34.1937471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1937544Z outputs = self.mobilebert( 2025-12-04T09:43:34.1937838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1937919Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1938214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1938290Z layer_outputs = layer_module( 2025-12-04T09:43:34.1938597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1938694Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1939000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1939115Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1939411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1939533Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1939537Z 2025-12-04T09:43:34.1939647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1939846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1939906Z res = mod(**inputs) 2025-12-04T09:43:34.1940194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1940270Z outputs = self.mobilebert( 2025-12-04T09:43:34.1940544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1940611Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1940889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1940956Z layer_outputs = layer_module( 2025-12-04T09:43:34.1941235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1941343Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1941635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1941768Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1942036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1942123Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1942126Z 2025-12-04T09:43:34.1942225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1942410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1942481Z res = mod(**inputs) 2025-12-04T09:43:34.1942749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1942841Z outputs = self.mobilebert( 2025-12-04T09:43:34.1943113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1943182Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1943460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1943529Z layer_outputs = layer_module( 2025-12-04T09:43:34.1943797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1943894Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1944166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1944291Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1944567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1944687Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1944966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1945054Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1945057Z 2025-12-04T09:43:34.1945163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1945355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1945415Z res = mod(**inputs) 2025-12-04T09:43:34.1945690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1945762Z outputs = self.mobilebert( 2025-12-04T09:43:34.1946035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1946114Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1946402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1946479Z layer_outputs = layer_module( 2025-12-04T09:43:34.1946747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1946832Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1947109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1947215Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1947507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1947603Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1947606Z 2025-12-04T09:43:34.1947706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1947901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1947967Z res = mod(**inputs) 2025-12-04T09:43:34.1948242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1948320Z outputs = self.mobilebert( 2025-12-04T09:43:34.1948595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1948673Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1948950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1949040Z layer_outputs = layer_module( 2025-12-04T09:43:34.1949324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1949414Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1949694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1949801Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1950087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1950202Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1950205Z 2025-12-04T09:43:34.1950303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1950504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1950566Z res = mod(**inputs) 2025-12-04T09:43:34.1950836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1950912Z outputs = self.mobilebert( 2025-12-04T09:43:34.1951179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1951247Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1951527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1951594Z layer_outputs = layer_module( 2025-12-04T09:43:34.1951872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1951962Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1952234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1952375Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1952644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1952731Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1952734Z 2025-12-04T09:43:34.1952831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1953025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1953092Z res = mod(**inputs) 2025-12-04T09:43:34.1953370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1953461Z outputs = self.mobilebert( 2025-12-04T09:43:34.1953770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1953842Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1954124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1954194Z layer_outputs = layer_module( 2025-12-04T09:43:34.1954472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1954567Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1954841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1954972Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1955270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1955390Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1955672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1955762Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1955765Z 2025-12-04T09:43:34.1955871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1956063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1956124Z res = mod(**inputs) 2025-12-04T09:43:34.1956413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1956483Z outputs = self.mobilebert( 2025-12-04T09:43:34.1956773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1956858Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1957148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1957228Z layer_outputs = layer_module( 2025-12-04T09:43:34.1957520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1957615Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1957916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1958029Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1958334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1958421Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1958448Z 2025-12-04T09:43:34.1958566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1958766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1958829Z res = mod(**inputs) 2025-12-04T09:43:34.1959106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1959183Z outputs = self.mobilebert( 2025-12-04T09:43:34.1959461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1959537Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1959832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1959917Z layer_outputs = layer_module( 2025-12-04T09:43:34.1960206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1960296Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1960578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.1960683Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.1960956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1961070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1961074Z 2025-12-04T09:43:34.1961175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1961383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1961454Z res = mod(**inputs) 2025-12-04T09:43:34.1961732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1961809Z outputs = self.mobilebert( 2025-12-04T09:43:34.1962088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1962161Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1962460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1962531Z layer_outputs = layer_module( 2025-12-04T09:43:34.1962828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1962927Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1963220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1963356Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1963643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.1963723Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1963733Z 2025-12-04T09:43:34.1963832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1964023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1964093Z res = mod(**inputs) 2025-12-04T09:43:34.1964367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1964439Z outputs = self.mobilebert( 2025-12-04T09:43:34.1964739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1964811Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1965093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1965162Z layer_outputs = layer_module( 2025-12-04T09:43:34.1965441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.1965539Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.1965817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.1965971Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.1966266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.1966388Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1966673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1966762Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1966765Z 2025-12-04T09:43:34.1966866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1967077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1967141Z res = mod(**inputs) 2025-12-04T09:43:34.1967441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1967535Z outputs = self.mobilebert( 2025-12-04T09:43:34.1967831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1967913Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1968210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1968288Z layer_outputs = layer_module( 2025-12-04T09:43:34.1968584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1968708Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1969007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.1969091Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.1969097Z 2025-12-04T09:43:34.1969200Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1969416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1969480Z res = mod(**inputs) 2025-12-04T09:43:34.1969779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1969849Z outputs = self.mobilebert( 2025-12-04T09:43:34.1970142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1970222Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1970519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1970597Z layer_outputs = layer_module( 2025-12-04T09:43:34.1970893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.1971020Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.1971427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.1971554Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.1971559Z 2025-12-04T09:43:34.1971677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1971887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1971954Z res = mod(**inputs) 2025-12-04T09:43:34.1972388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1972482Z outputs = self.mobilebert( 2025-12-04T09:43:34.1972823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1972936Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1973241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1973322Z layer_outputs = layer_module( 2025-12-04T09:43:34.1973623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1973790Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1974099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.1974198Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.1974229Z 2025-12-04T09:43:34.1974352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1974567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1974635Z res = mod(**inputs) 2025-12-04T09:43:34.1974957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1975033Z outputs = self.mobilebert( 2025-12-04T09:43:34.1975350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1975434Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1975752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1975837Z layer_outputs = layer_module( 2025-12-04T09:43:34.1976152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1976327Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1976656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.1976788Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.1977112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1977209Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1977213Z 2025-12-04T09:43:34.1977324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1977546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1977611Z res = mod(**inputs) 2025-12-04T09:43:34.1977928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1978010Z outputs = self.mobilebert( 2025-12-04T09:43:34.1978361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1978448Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1978766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1978839Z layer_outputs = layer_module( 2025-12-04T09:43:34.1979161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1979328Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1979647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1979797Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1980130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.1980229Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.1980233Z 2025-12-04T09:43:34.1980343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1980578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1980645Z res = mod(**inputs) 2025-12-04T09:43:34.1980958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1981038Z outputs = self.mobilebert( 2025-12-04T09:43:34.1981350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1981446Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1981777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1981853Z layer_outputs = layer_module( 2025-12-04T09:43:34.1982181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.1982349Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.1982673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.1982814Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.1983132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.1983275Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.1983599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1983697Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1983701Z 2025-12-04T09:43:34.1983818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1984044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1984118Z res = mod(**inputs) 2025-12-04T09:43:34.1984438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1984512Z outputs = self.mobilebert( 2025-12-04T09:43:34.1984840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1984917Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1985258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1985354Z layer_outputs = layer_module( 2025-12-04T09:43:34.1985649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1985822Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1986117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1986233Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1986545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1986655Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1986659Z 2025-12-04T09:43:34.1986790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1986993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1987061Z res = mod(**inputs) 2025-12-04T09:43:34.1987358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1987432Z outputs = self.mobilebert( 2025-12-04T09:43:34.1987723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1987802Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1988092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1988199Z layer_outputs = layer_module( 2025-12-04T09:43:34.1988502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1988591Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1988899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1988972Z self_outputs = self.self( 2025-12-04T09:43:34.1989280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.1989352Z self.value(value_tensor) 2025-12-04T09:43:34.1989356Z 2025-12-04T09:43:34.1989461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1989671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1989740Z res = mod(**inputs) 2025-12-04T09:43:34.1990044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1990125Z outputs = self.mobilebert( 2025-12-04T09:43:34.1990422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1990502Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1990811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1990882Z layer_outputs = layer_module( 2025-12-04T09:43:34.1991189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1991354Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1991671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.1991785Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.1992114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.1992210Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.1992213Z 2025-12-04T09:43:34.1992318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1992531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1992598Z res = mod(**inputs) 2025-12-04T09:43:34.1992891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1992973Z outputs = self.mobilebert( 2025-12-04T09:43:34.1993286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1993383Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1993688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1993761Z layer_outputs = layer_module( 2025-12-04T09:43:34.1994064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.1994231Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.1994526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.1994649Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.1994973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.1995067Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.1995371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.1995467Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.1995471Z 2025-12-04T09:43:34.1995584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1995789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1995853Z res = mod(**inputs) 2025-12-04T09:43:34.1996160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1996228Z outputs = self.mobilebert( 2025-12-04T09:43:34.1996511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1996582Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1996857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1996934Z layer_outputs = layer_module( 2025-12-04T09:43:34.1997212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1997302Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.1997575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.1997644Z self_outputs = self.self( 2025-12-04T09:43:34.1997928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.1998000Z self.query(query_tensor) 2025-12-04T09:43:34.1998003Z 2025-12-04T09:43:34.1998105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.1998321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.1998384Z res = mod(**inputs) 2025-12-04T09:43:34.1998670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.1998736Z outputs = self.mobilebert( 2025-12-04T09:43:34.1999010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.1999088Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.1999364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.1999461Z layer_outputs = layer_module( 2025-12-04T09:43:34.1999754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.1999840Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2000123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2000192Z self_outputs = self.self( 2025-12-04T09:43:34.2000465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.2000539Z self.key(key_tensor) 2025-12-04T09:43:34.2000542Z 2025-12-04T09:43:34.2000621Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2000706Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2000807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2001002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2001089Z res = mod(**inputs) 2025-12-04T09:43:34.2001388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2001457Z outputs = self.mobilebert( 2025-12-04T09:43:34.2001751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2001822Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2002113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2002180Z layer_outputs = layer_module( 2025-12-04T09:43:34.2002465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2002557Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2002848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2002977Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2003268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.2003351Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2003354Z 2025-12-04T09:43:34.2003460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2003660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2003722Z res = mod(**inputs) 2025-12-04T09:43:34.2004016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2004087Z outputs = self.mobilebert( 2025-12-04T09:43:34.2004382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2004453Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2004759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2004837Z layer_outputs = layer_module( 2025-12-04T09:43:34.2005114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2005200Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2005480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2005600Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2005905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.2006047Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2006335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2006426Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2006429Z 2025-12-04T09:43:34.2006531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2006731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2006794Z res = mod(**inputs) 2025-12-04T09:43:34.2007070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2007147Z outputs = self.mobilebert( 2025-12-04T09:43:34.2007439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2007520Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2007799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2007867Z layer_outputs = layer_module( 2025-12-04T09:43:34.2008152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2008243Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2008527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2008636Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2008913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2009005Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2009008Z 2025-12-04T09:43:34.2009112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2009316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2009390Z res = mod(**inputs) 2025-12-04T09:43:34.2009683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2009766Z outputs = self.mobilebert( 2025-12-04T09:43:34.2010059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2010133Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2010433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2010509Z layer_outputs = layer_module( 2025-12-04T09:43:34.2010828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2010927Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2011286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2011422Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2011721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2011841Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2011851Z 2025-12-04T09:43:34.2011965Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2012210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2012288Z res = mod(**inputs) 2025-12-04T09:43:34.2012618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2012697Z outputs = self.mobilebert( 2025-12-04T09:43:34.2013016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2013094Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2013412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2013485Z layer_outputs = layer_module( 2025-12-04T09:43:34.2013790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2013925Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2014222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2014354Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2014658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2014743Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2014747Z 2025-12-04T09:43:34.2014857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2015061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2015124Z res = mod(**inputs) 2025-12-04T09:43:34.2015423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2015497Z outputs = self.mobilebert( 2025-12-04T09:43:34.2015801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2015875Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2016166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2016246Z layer_outputs = layer_module( 2025-12-04T09:43:34.2016536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2016638Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2016930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2017058Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2017365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2017507Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2017805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2017907Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2017911Z 2025-12-04T09:43:34.2018015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2018226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2018292Z res = mod(**inputs) 2025-12-04T09:43:34.2018653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2018754Z outputs = self.mobilebert( 2025-12-04T09:43:34.2019053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2019150Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2019445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2019517Z layer_outputs = layer_module( 2025-12-04T09:43:34.2019883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2019981Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2020275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2020396Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2020691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2020809Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2020813Z 2025-12-04T09:43:34.2020920Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2021126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2021199Z res = mod(**inputs) 2025-12-04T09:43:34.2021493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2021570Z outputs = self.mobilebert( 2025-12-04T09:43:34.2021872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2021944Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2022252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2022323Z layer_outputs = layer_module( 2025-12-04T09:43:34.2022596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2022690Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2022960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2023071Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2023343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2023448Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2023451Z 2025-12-04T09:43:34.2023554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2023747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2023813Z res = mod(**inputs) 2025-12-04T09:43:34.2024107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2024176Z outputs = self.mobilebert( 2025-12-04T09:43:34.2024457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2024526Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2024803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2024870Z layer_outputs = layer_module( 2025-12-04T09:43:34.2025147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2025260Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2025555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2025677Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2025957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2026036Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2026039Z 2025-12-04T09:43:34.2026144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2026335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2026396Z res = mod(**inputs) 2025-12-04T09:43:34.2026676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2026769Z outputs = self.mobilebert( 2025-12-04T09:43:34.2027096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2027218Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2027515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2027644Z layer_outputs = layer_module( 2025-12-04T09:43:34.2027935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2028270Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2028568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2028741Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2029051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2029226Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2029525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2029657Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2029661Z 2025-12-04T09:43:34.2029824Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2030039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2030125Z res = mod(**inputs) 2025-12-04T09:43:34.2030443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2030526Z outputs = self.mobilebert( 2025-12-04T09:43:34.2030876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2030992Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2031281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2031399Z layer_outputs = layer_module( 2025-12-04T09:43:34.2031694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2031834Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2032152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2032285Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2032622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2032736Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2032740Z 2025-12-04T09:43:34.2032878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2033103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2033203Z res = mod(**inputs) 2025-12-04T09:43:34.2033526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2033616Z outputs = self.mobilebert( 2025-12-04T09:43:34.2033937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2034017Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2034340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2034479Z layer_outputs = layer_module( 2025-12-04T09:43:34.2034774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2034883Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2035203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2035326Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2035670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2035800Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2035804Z 2025-12-04T09:43:34.2035924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2036169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2036261Z res = mod(**inputs) 2025-12-04T09:43:34.2036641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2036740Z outputs = self.mobilebert( 2025-12-04T09:43:34.2037033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2037149Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2037436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2037545Z layer_outputs = layer_module( 2025-12-04T09:43:34.2037857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2037981Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2038316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2038468Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2038781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2038868Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2038872Z 2025-12-04T09:43:34.2039005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2039248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2039329Z res = mod(**inputs) 2025-12-04T09:43:34.2039617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2039749Z outputs = self.mobilebert( 2025-12-04T09:43:34.2040038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2040175Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2040456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2040549Z layer_outputs = layer_module( 2025-12-04T09:43:34.2040884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2040990Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2041306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2041467Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2041756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2041919Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2042200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2042322Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2042325Z 2025-12-04T09:43:34.2042459Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2042680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2042784Z res = mod(**inputs) 2025-12-04T09:43:34.2043065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2043184Z outputs = self.mobilebert( 2025-12-04T09:43:34.2043457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2043561Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2043893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2043979Z layer_outputs = layer_module( 2025-12-04T09:43:34.2044262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2044423Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2044698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2044856Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2044861Z 2025-12-04T09:43:34.2044977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2045181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2045325Z res = mod(**inputs) 2025-12-04T09:43:34.2045608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2067796Z outputs = self.mobilebert( 2025-12-04T09:43:34.2068265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2068361Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2068663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2068745Z layer_outputs = layer_module( 2025-12-04T09:43:34.2069174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2069342Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2069691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2069809Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2069816Z 2025-12-04T09:43:34.2069934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2070140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2070214Z res = mod(**inputs) 2025-12-04T09:43:34.2070488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2070566Z outputs = self.mobilebert( 2025-12-04T09:43:34.2070883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2070960Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2071237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2071314Z layer_outputs = layer_module( 2025-12-04T09:43:34.2071589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2071754Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2072027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.2072122Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.2072128Z 2025-12-04T09:43:34.2072244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2072696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2072775Z res = mod(**inputs) 2025-12-04T09:43:34.2073052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2073127Z outputs = self.mobilebert( 2025-12-04T09:43:34.2073405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2073476Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2073755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2073827Z layer_outputs = layer_module( 2025-12-04T09:43:34.2074098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2074266Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2074596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.2074726Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.2075009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2075104Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2075108Z 2025-12-04T09:43:34.2075218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2075411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2075474Z res = mod(**inputs) 2025-12-04T09:43:34.2075785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2075882Z outputs = self.mobilebert( 2025-12-04T09:43:34.2076166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2076237Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2076506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2076584Z layer_outputs = layer_module( 2025-12-04T09:43:34.2076858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2077007Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2077289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2077438Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2077720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.2077802Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2077806Z 2025-12-04T09:43:34.2077906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2078109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2078172Z res = mod(**inputs) 2025-12-04T09:43:34.2078450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2078520Z outputs = self.mobilebert( 2025-12-04T09:43:34.2078791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2078870Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2079142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2079217Z layer_outputs = layer_module( 2025-12-04T09:43:34.2079486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2079631Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2079910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2080024Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2080296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.2080423Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2080713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2080813Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2080817Z 2025-12-04T09:43:34.2080915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2081108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2081179Z res = mod(**inputs) 2025-12-04T09:43:34.2081457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2081533Z outputs = self.mobilebert( 2025-12-04T09:43:34.2081813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2081902Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2082209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2082279Z layer_outputs = layer_module( 2025-12-04T09:43:34.2082552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2082716Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2082993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.2083111Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.2083387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.2083486Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.2083490Z 2025-12-04T09:43:34.2083596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2083789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2083857Z res = mod(**inputs) 2025-12-04T09:43:34.2084134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2084201Z outputs = self.mobilebert( 2025-12-04T09:43:34.2084485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2084551Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2084836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2084908Z layer_outputs = layer_module( 2025-12-04T09:43:34.2085188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2085282Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2085558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2085626Z self_outputs = self.self( 2025-12-04T09:43:34.2085924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.2085995Z self.value(value_tensor) 2025-12-04T09:43:34.2085998Z 2025-12-04T09:43:34.2086108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2086308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2086372Z res = mod(**inputs) 2025-12-04T09:43:34.2086664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2086748Z outputs = self.mobilebert( 2025-12-04T09:43:34.2087037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2087108Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2087384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2087460Z layer_outputs = layer_module( 2025-12-04T09:43:34.2087739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2087898Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2088213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.2088340Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.2088635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.2088720Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.2088724Z 2025-12-04T09:43:34.2088833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2089050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2089115Z res = mod(**inputs) 2025-12-04T09:43:34.2089420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2089493Z outputs = self.mobilebert( 2025-12-04T09:43:34.2089808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2089894Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2090192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2090268Z layer_outputs = layer_module( 2025-12-04T09:43:34.2090576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2090733Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2091019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.2091127Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.2091474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.2091578Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.2091874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2091979Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2091983Z 2025-12-04T09:43:34.2092090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2092298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2092375Z res = mod(**inputs) 2025-12-04T09:43:34.2092673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2092758Z outputs = self.mobilebert( 2025-12-04T09:43:34.2093056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2093134Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2093475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2093552Z layer_outputs = layer_module( 2025-12-04T09:43:34.2093853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2093954Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2094255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2094336Z self_outputs = self.self( 2025-12-04T09:43:34.2094639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.2094733Z self.query(query_tensor) 2025-12-04T09:43:34.2094737Z 2025-12-04T09:43:34.2094852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2095079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2095148Z res = mod(**inputs) 2025-12-04T09:43:34.2095426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2095495Z outputs = self.mobilebert( 2025-12-04T09:43:34.2095779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2095848Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2096124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2096234Z layer_outputs = layer_module( 2025-12-04T09:43:34.2096513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2096603Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2096882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2096952Z self_outputs = self.self( 2025-12-04T09:43:34.2097238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.2097305Z self.key(key_tensor) 2025-12-04T09:43:34.2097308Z 2025-12-04T09:43:34.2097399Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2097478Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2097581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2097786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2097849Z res = mod(**inputs) 2025-12-04T09:43:34.2098142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2098210Z outputs = self.mobilebert( 2025-12-04T09:43:34.2098488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2098571Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2098865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2098946Z layer_outputs = layer_module( 2025-12-04T09:43:34.2099245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2099330Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2099620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2099761Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2100049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.2100133Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2100137Z 2025-12-04T09:43:34.2100240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2100440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2100502Z res = mod(**inputs) 2025-12-04T09:43:34.2100782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2100879Z outputs = self.mobilebert( 2025-12-04T09:43:34.2101169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2101261Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2101540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2101609Z layer_outputs = layer_module( 2025-12-04T09:43:34.2101891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2101970Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2102254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2102376Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2102681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.2102813Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2103092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2103182Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2103194Z 2025-12-04T09:43:34.2103294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2103490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2103559Z res = mod(**inputs) 2025-12-04T09:43:34.2103836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2103904Z outputs = self.mobilebert( 2025-12-04T09:43:34.2104192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2104262Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2104545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2104613Z layer_outputs = layer_module( 2025-12-04T09:43:34.2104889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2104990Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2105268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2105381Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2105666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2105750Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2105754Z 2025-12-04T09:43:34.2105862Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2106073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2106138Z res = mod(**inputs) 2025-12-04T09:43:34.2106434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2106502Z outputs = self.mobilebert( 2025-12-04T09:43:34.2106792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2106860Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2107145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2107243Z layer_outputs = layer_module( 2025-12-04T09:43:34.2107540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2107643Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2107927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2108037Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2108325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2108437Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2108441Z 2025-12-04T09:43:34.2108540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2108747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2108821Z res = mod(**inputs) 2025-12-04T09:43:34.2109105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2109170Z outputs = self.mobilebert( 2025-12-04T09:43:34.2109444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2109519Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2109795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2109866Z layer_outputs = layer_module( 2025-12-04T09:43:34.2110148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2110238Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2110523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2110645Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2110924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2111008Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2111011Z 2025-12-04T09:43:34.2111107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2111305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2111362Z res = mod(**inputs) 2025-12-04T09:43:34.2111636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2111710Z outputs = self.mobilebert( 2025-12-04T09:43:34.2111986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2112096Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2112362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2112427Z layer_outputs = layer_module( 2025-12-04T09:43:34.2112696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2112782Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2113045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2113168Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2113449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2113587Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2113856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2113943Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2113946Z 2025-12-04T09:43:34.2114048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2114232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2114299Z res = mod(**inputs) 2025-12-04T09:43:34.2114564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2114631Z outputs = self.mobilebert( 2025-12-04T09:43:34.2114927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2114995Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2115271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2115336Z layer_outputs = layer_module( 2025-12-04T09:43:34.2115598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2115693Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2115955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2116058Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2116328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2116407Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2116412Z 2025-12-04T09:43:34.2116515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2116696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2116755Z res = mod(**inputs) 2025-12-04T09:43:34.2117023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2117089Z outputs = self.mobilebert( 2025-12-04T09:43:34.2117362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2117428Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2117687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2117761Z layer_outputs = layer_module( 2025-12-04T09:43:34.2118038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2118124Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2118397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2118500Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2118771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2118877Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2118881Z 2025-12-04T09:43:34.2118978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2119189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2119250Z res = mod(**inputs) 2025-12-04T09:43:34.2119539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2119607Z outputs = self.mobilebert( 2025-12-04T09:43:34.2119868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2119943Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2120206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2120272Z layer_outputs = layer_module( 2025-12-04T09:43:34.2120540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2120645Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2120924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2121039Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2121306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2121389Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2121393Z 2025-12-04T09:43:34.2121489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2121681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2121740Z res = mod(**inputs) 2025-12-04T09:43:34.2122006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2122085Z outputs = self.mobilebert( 2025-12-04T09:43:34.2122362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2122436Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2122715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2122781Z layer_outputs = layer_module( 2025-12-04T09:43:34.2123055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2123150Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2123426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2123549Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2123833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2123964Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2124243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2124330Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2124334Z 2025-12-04T09:43:34.2124435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2124628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2124688Z res = mod(**inputs) 2025-12-04T09:43:34.2124966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2125054Z outputs = self.mobilebert( 2025-12-04T09:43:34.2125353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2125436Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2125715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2125791Z layer_outputs = layer_module( 2025-12-04T09:43:34.2126072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2126162Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2126447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2126556Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2126856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2126951Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2126955Z 2025-12-04T09:43:34.2127061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2127268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2127332Z res = mod(**inputs) 2025-12-04T09:43:34.2127624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2127704Z outputs = self.mobilebert( 2025-12-04T09:43:34.2127994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2128071Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2128365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2128438Z layer_outputs = layer_module( 2025-12-04T09:43:34.2128740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2128833Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2129126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2129247Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2129540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2129660Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2129663Z 2025-12-04T09:43:34.2129772Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2129980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2130054Z res = mod(**inputs) 2025-12-04T09:43:34.2130365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2130447Z outputs = self.mobilebert( 2025-12-04T09:43:34.2130741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2130814Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2131113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2131185Z layer_outputs = layer_module( 2025-12-04T09:43:34.2131555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2131688Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2132002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2132141Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2132437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2132524Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2132528Z 2025-12-04T09:43:34.2132644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2132850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2132924Z res = mod(**inputs) 2025-12-04T09:43:34.2133218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2133317Z outputs = self.mobilebert( 2025-12-04T09:43:34.2133627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2133709Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2133994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2134063Z layer_outputs = layer_module( 2025-12-04T09:43:34.2134340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2134437Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2134713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2134836Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2135123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2135244Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2135527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2135615Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2135618Z 2025-12-04T09:43:34.2135719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2135918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2135981Z res = mod(**inputs) 2025-12-04T09:43:34.2136265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2136346Z outputs = self.mobilebert( 2025-12-04T09:43:34.2136616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2136709Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2136979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2137047Z layer_outputs = layer_module( 2025-12-04T09:43:34.2137322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2137441Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2137730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2137826Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2137832Z 2025-12-04T09:43:34.2137930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2138158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2138223Z res = mod(**inputs) 2025-12-04T09:43:34.2138504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2138571Z outputs = self.mobilebert( 2025-12-04T09:43:34.2138840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2138916Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2139186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2139253Z layer_outputs = layer_module( 2025-12-04T09:43:34.2139557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2139676Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2139971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2140088Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2140092Z 2025-12-04T09:43:34.2140191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2140391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2140450Z res = mod(**inputs) 2025-12-04T09:43:34.2140738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2140805Z outputs = self.mobilebert( 2025-12-04T09:43:34.2141086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2141163Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2141453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2141529Z layer_outputs = layer_module( 2025-12-04T09:43:34.2141817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2141977Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2142268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.2142358Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.2142363Z 2025-12-04T09:43:34.2142465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2142671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2142735Z res = mod(**inputs) 2025-12-04T09:43:34.2143041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2143111Z outputs = self.mobilebert( 2025-12-04T09:43:34.2143388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2143467Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2143751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2143825Z layer_outputs = layer_module( 2025-12-04T09:43:34.2144103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2144276Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2144584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.2144706Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.2144984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2145080Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2145083Z 2025-12-04T09:43:34.2145183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2145381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2145441Z res = mod(**inputs) 2025-12-04T09:43:34.2145738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2145814Z outputs = self.mobilebert( 2025-12-04T09:43:34.2146091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2146166Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2146438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2146506Z layer_outputs = layer_module( 2025-12-04T09:43:34.2146784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2146934Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2147217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2147339Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2147616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.2147704Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2147708Z 2025-12-04T09:43:34.2147807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2147999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2148069Z res = mod(**inputs) 2025-12-04T09:43:34.2148341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2148418Z outputs = self.mobilebert( 2025-12-04T09:43:34.2148694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2148767Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2149069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2149139Z layer_outputs = layer_module( 2025-12-04T09:43:34.2149422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2149572Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2149846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2149974Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2150253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.2150390Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2150691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2150783Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2150787Z 2025-12-04T09:43:34.2150894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2151089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2151152Z res = mod(**inputs) 2025-12-04T09:43:34.2151437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2151506Z outputs = self.mobilebert( 2025-12-04T09:43:34.2151789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2151880Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2152163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2152240Z layer_outputs = layer_module( 2025-12-04T09:43:34.2152519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2152689Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2152970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.2153080Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.2153369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.2153453Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.2153457Z 2025-12-04T09:43:34.2153558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2153762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2153825Z res = mod(**inputs) 2025-12-04T09:43:34.2154112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2154183Z outputs = self.mobilebert( 2025-12-04T09:43:34.2154467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2154544Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2154822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2154903Z layer_outputs = layer_module( 2025-12-04T09:43:34.2155183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2155283Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2155568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2155637Z self_outputs = self.self( 2025-12-04T09:43:34.2155919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.2155996Z self.value(value_tensor) 2025-12-04T09:43:34.2156000Z 2025-12-04T09:43:34.2156099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2156300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2156415Z res = mod(**inputs) 2025-12-04T09:43:34.2156726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2156808Z outputs = self.mobilebert( 2025-12-04T09:43:34.2157107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2157187Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2157483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2157555Z layer_outputs = layer_module( 2025-12-04T09:43:34.2157859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2158023Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2158339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.2158464Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.2158759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.2158850Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.2158854Z 2025-12-04T09:43:34.2158959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2159164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2159236Z res = mod(**inputs) 2025-12-04T09:43:34.2159528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2159606Z outputs = self.mobilebert( 2025-12-04T09:43:34.2159903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2159992Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2160287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2160357Z layer_outputs = layer_module( 2025-12-04T09:43:34.2160655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2160818Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2161112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.2161230Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.2161523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.2161622Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.2161934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2162032Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2162036Z 2025-12-04T09:43:34.2162150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2162359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2162433Z res = mod(**inputs) 2025-12-04T09:43:34.2162731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2162803Z outputs = self.mobilebert( 2025-12-04T09:43:34.2163107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2163209Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2163540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2163614Z layer_outputs = layer_module( 2025-12-04T09:43:34.2163908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2164005Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2164307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2164380Z self_outputs = self.self( 2025-12-04T09:43:34.2164693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.2164789Z self.query(query_tensor) 2025-12-04T09:43:34.2164794Z 2025-12-04T09:43:34.2164907Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2165119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2165184Z res = mod(**inputs) 2025-12-04T09:43:34.2165481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2165553Z outputs = self.mobilebert( 2025-12-04T09:43:34.2165855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2165927Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2166221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2166301Z layer_outputs = layer_module( 2025-12-04T09:43:34.2166598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2166686Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2167041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2167113Z self_outputs = self.self( 2025-12-04T09:43:34.2167426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.2167496Z self.key(key_tensor) 2025-12-04T09:43:34.2167500Z 2025-12-04T09:43:34.2167586Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2167677Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2167786Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2168011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2168088Z res = mod(**inputs) 2025-12-04T09:43:34.2168414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2168498Z outputs = self.mobilebert( 2025-12-04T09:43:34.2168813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2168887Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2169207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2169279Z layer_outputs = layer_module( 2025-12-04T09:43:34.2169600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2169706Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2170019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2170176Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2170492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.2170584Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2170595Z 2025-12-04T09:43:34.2170706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2170916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2170989Z res = mod(**inputs) 2025-12-04T09:43:34.2171483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2171569Z outputs = self.mobilebert( 2025-12-04T09:43:34.2171917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2171995Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2172458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2172542Z layer_outputs = layer_module( 2025-12-04T09:43:34.2172857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2172953Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2173249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2173375Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2173675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.2173809Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2174109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2174204Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2174208Z 2025-12-04T09:43:34.2174314Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2174527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2174593Z res = mod(**inputs) 2025-12-04T09:43:34.2174893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2174967Z outputs = self.mobilebert( 2025-12-04T09:43:34.2175265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2175349Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2175690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2175771Z layer_outputs = layer_module( 2025-12-04T09:43:34.2176065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2176163Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2176464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2176579Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2176871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2176995Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2176999Z 2025-12-04T09:43:34.2177133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2177353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2177422Z res = mod(**inputs) 2025-12-04T09:43:34.2177716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2177802Z outputs = self.mobilebert( 2025-12-04T09:43:34.2178097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2178178Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2178473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2178575Z layer_outputs = layer_module( 2025-12-04T09:43:34.2178895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2178986Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2179261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2179378Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2179655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2179772Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2179776Z 2025-12-04T09:43:34.2179878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2180075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2180148Z res = mod(**inputs) 2025-12-04T09:43:34.2180428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2180504Z outputs = self.mobilebert( 2025-12-04T09:43:34.2180779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2180847Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2181130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2181201Z layer_outputs = layer_module( 2025-12-04T09:43:34.2181483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2181584Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2181855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2182002Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2182272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2182352Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2182355Z 2025-12-04T09:43:34.2182460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2182667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2182740Z res = mod(**inputs) 2025-12-04T09:43:34.2183053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2183160Z outputs = self.mobilebert( 2025-12-04T09:43:34.2183481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2183559Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2183854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2183933Z layer_outputs = layer_module( 2025-12-04T09:43:34.2184222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2184323Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2184624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2184757Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2185084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2185225Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2185525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2185620Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2185623Z 2025-12-04T09:43:34.2185728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2185939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2186005Z res = mod(**inputs) 2025-12-04T09:43:34.2186302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2186375Z outputs = self.mobilebert( 2025-12-04T09:43:34.2186669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2186745Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2187021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2187089Z layer_outputs = layer_module( 2025-12-04T09:43:34.2187377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2187464Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2187736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2187838Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2188107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2188194Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2188199Z 2025-12-04T09:43:34.2188320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2188531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2188594Z res = mod(**inputs) 2025-12-04T09:43:34.2188868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2188945Z outputs = self.mobilebert( 2025-12-04T09:43:34.2189222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2189291Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2189578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2189664Z layer_outputs = layer_module( 2025-12-04T09:43:34.2189981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2190075Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2190353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2190470Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2190747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2190863Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2190866Z 2025-12-04T09:43:34.2190966Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2191178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2191249Z res = mod(**inputs) 2025-12-04T09:43:34.2191542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2191618Z outputs = self.mobilebert( 2025-12-04T09:43:34.2191894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2191969Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2192280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2192355Z layer_outputs = layer_module( 2025-12-04T09:43:34.2192655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2192765Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2193068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2193207Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2193507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2193596Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2193600Z 2025-12-04T09:43:34.2193716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2193930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2194006Z res = mod(**inputs) 2025-12-04T09:43:34.2194311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2194386Z outputs = self.mobilebert( 2025-12-04T09:43:34.2194685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2194775Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2195070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2195151Z layer_outputs = layer_module( 2025-12-04T09:43:34.2195441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2195545Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2195839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2195967Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2196304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2196434Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2196733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2196829Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2196833Z 2025-12-04T09:43:34.2196937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2197151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2197218Z res = mod(**inputs) 2025-12-04T09:43:34.2197517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2197623Z outputs = self.mobilebert( 2025-12-04T09:43:34.2197937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2198020Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2198318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2198392Z layer_outputs = layer_module( 2025-12-04T09:43:34.2198696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2198792Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2199094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2199209Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2199506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2199601Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2199605Z 2025-12-04T09:43:34.2199711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2199923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2199989Z res = mod(**inputs) 2025-12-04T09:43:34.2200281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2200363Z outputs = self.mobilebert( 2025-12-04T09:43:34.2200666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2200741Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2201054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2201130Z layer_outputs = layer_module( 2025-12-04T09:43:34.2201458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2201559Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2201861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2201985Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2202292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2202416Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2202420Z 2025-12-04T09:43:34.2202548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2202762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2202864Z res = mod(**inputs) 2025-12-04T09:43:34.2203162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2203237Z outputs = self.mobilebert( 2025-12-04T09:43:34.2203539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2203613Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2203918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2203988Z layer_outputs = layer_module( 2025-12-04T09:43:34.2204282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2204404Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2204696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2204831Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2205125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2205208Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2205211Z 2025-12-04T09:43:34.2205325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2205530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2205595Z res = mod(**inputs) 2025-12-04T09:43:34.2205894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2205973Z outputs = self.mobilebert( 2025-12-04T09:43:34.2206277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2206351Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2206647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2206730Z layer_outputs = layer_module( 2025-12-04T09:43:34.2207033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2207550Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2208031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2208555Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2209053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2209560Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2210057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2210525Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2210701Z 2025-12-04T09:43:34.2210816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2211297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2211678Z res = mod(**inputs) 2025-12-04T09:43:34.2212105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2212590Z outputs = self.mobilebert( 2025-12-04T09:43:34.2213061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2213519Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2214051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2214520Z layer_outputs = layer_module( 2025-12-04T09:43:34.2214982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2215481Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2215974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2216451Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2216647Z 2025-12-04T09:43:34.2216758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2217151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2217496Z res = mod(**inputs) 2025-12-04T09:43:34.2217917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2218388Z outputs = self.mobilebert( 2025-12-04T09:43:34.2218824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2219264Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2219726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2221098Z layer_outputs = layer_module( 2025-12-04T09:43:34.2221551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2222075Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2222579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2223072Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2223253Z 2025-12-04T09:43:34.2223356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2223718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2224044Z res = mod(**inputs) 2025-12-04T09:43:34.2224433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2224852Z outputs = self.mobilebert( 2025-12-04T09:43:34.2225254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2225672Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2226097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2226511Z layer_outputs = layer_module( 2025-12-04T09:43:34.2226929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2227454Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2227974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.2228433Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.2228613Z 2025-12-04T09:43:34.2228732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2229094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2229595Z res = mod(**inputs) 2025-12-04T09:43:34.2230119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2230542Z outputs = self.mobilebert( 2025-12-04T09:43:34.2230934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2231361Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2231786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2232214Z layer_outputs = layer_module( 2025-12-04T09:43:34.2232628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2233190Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2233722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.2234202Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.2234692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2235150Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2235307Z 2025-12-04T09:43:34.2235427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2235800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2236142Z res = mod(**inputs) 2025-12-04T09:43:34.2236555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2236993Z outputs = self.mobilebert( 2025-12-04T09:43:34.2237416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2237864Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2238358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2238795Z layer_outputs = layer_module( 2025-12-04T09:43:34.2239298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2239822Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2240352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2240838Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2241349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.2241798Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2241944Z 2025-12-04T09:43:34.2242058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2242424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2242765Z res = mod(**inputs) 2025-12-04T09:43:34.2243172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2243608Z outputs = self.mobilebert( 2025-12-04T09:43:34.2244020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2244473Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2244912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2245343Z layer_outputs = layer_module( 2025-12-04T09:43:34.2245778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2246304Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2246833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2247317Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2247814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.2248326Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2248813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2249267Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2249426Z 2025-12-04T09:43:34.2249534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2249905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2250234Z res = mod(**inputs) 2025-12-04T09:43:34.2250639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2251071Z outputs = self.mobilebert( 2025-12-04T09:43:34.2251558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2252010Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2252452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2252899Z layer_outputs = layer_module( 2025-12-04T09:43:34.2253338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2253882Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2254429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.2254903Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.2255371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.2255825Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.2255981Z 2025-12-04T09:43:34.2256095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2256511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2256851Z res = mod(**inputs) 2025-12-04T09:43:34.2257261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2257695Z outputs = self.mobilebert( 2025-12-04T09:43:34.2258117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2258546Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2258976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2259433Z layer_outputs = layer_module( 2025-12-04T09:43:34.2259870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2260329Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2260776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2261210Z self_outputs = self.self( 2025-12-04T09:43:34.2261632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.2262066Z self.value(value_tensor) 2025-12-04T09:43:34.2262187Z 2025-12-04T09:43:34.2262304Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2262683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2263044Z res = mod(**inputs) 2025-12-04T09:43:34.2263451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2263892Z outputs = self.mobilebert( 2025-12-04T09:43:34.2264310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2264752Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2265185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2265623Z layer_outputs = layer_module( 2025-12-04T09:43:34.2266046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2266583Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2267125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.2267611Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.2268081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.2268540Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.2268688Z 2025-12-04T09:43:34.2268803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2269170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2269512Z res = mod(**inputs) 2025-12-04T09:43:34.2269921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2270366Z outputs = self.mobilebert( 2025-12-04T09:43:34.2270785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2271231Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2271684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2272127Z layer_outputs = layer_module( 2025-12-04T09:43:34.2272792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2273339Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2273876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.2274442Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.2274968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.2275426Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.2275926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2276387Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2276558Z 2025-12-04T09:43:34.2276669Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2277052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2277398Z res = mod(**inputs) 2025-12-04T09:43:34.2277809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2278252Z outputs = self.mobilebert( 2025-12-04T09:43:34.2278678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2279147Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2279577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2280009Z layer_outputs = layer_module( 2025-12-04T09:43:34.2280438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2280883Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2281335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2281750Z self_outputs = self.self( 2025-12-04T09:43:34.2282150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.2282557Z self.query(query_tensor) 2025-12-04T09:43:34.2282679Z 2025-12-04T09:43:34.2282783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2283156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2283487Z res = mod(**inputs) 2025-12-04T09:43:34.2283890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2284323Z outputs = self.mobilebert( 2025-12-04T09:43:34.2284743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2285174Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2285603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2286037Z layer_outputs = layer_module( 2025-12-04T09:43:34.2286464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2286912Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2287395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2287837Z self_outputs = self.self( 2025-12-04T09:43:34.2288263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.2288713Z self.key(key_tensor) 2025-12-04T09:43:34.2288841Z 2025-12-04T09:43:34.2288930Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2289161Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2289408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2289813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2290198Z res = mod(**inputs) 2025-12-04T09:43:34.2290633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2291107Z outputs = self.mobilebert( 2025-12-04T09:43:34.2291680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2292170Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2292618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2293054Z layer_outputs = layer_module( 2025-12-04T09:43:34.2293546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2294003Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2294490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2294983Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2295473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.2295919Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2296072Z 2025-12-04T09:43:34.2296181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2296556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2296897Z res = mod(**inputs) 2025-12-04T09:43:34.2297297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2297738Z outputs = self.mobilebert( 2025-12-04T09:43:34.2298169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2298609Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2299032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2299466Z layer_outputs = layer_module( 2025-12-04T09:43:34.2299901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2300358Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2300813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2301311Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2301808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.2302297Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2302825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2303292Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2303450Z 2025-12-04T09:43:34.2303570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2303941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2304283Z res = mod(**inputs) 2025-12-04T09:43:34.2304690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2305129Z outputs = self.mobilebert( 2025-12-04T09:43:34.2305590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2306043Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2306516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2306959Z layer_outputs = layer_module( 2025-12-04T09:43:34.2307420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2307888Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2308357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2308861Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2309335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2309823Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2309973Z 2025-12-04T09:43:34.2310082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2310460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2310797Z res = mod(**inputs) 2025-12-04T09:43:34.2311201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2311627Z outputs = self.mobilebert( 2025-12-04T09:43:34.2312057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2312496Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2312924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2313349Z layer_outputs = layer_module( 2025-12-04T09:43:34.2313775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2314234Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2314686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2315162Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2315657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2316130Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2316304Z 2025-12-04T09:43:34.2316412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2316790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2317131Z res = mod(**inputs) 2025-12-04T09:43:34.2317563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2317995Z outputs = self.mobilebert( 2025-12-04T09:43:34.2318418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2318850Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2319270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2319702Z layer_outputs = layer_module( 2025-12-04T09:43:34.2320129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2320608Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2321074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2321570Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2322063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2322509Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2322655Z 2025-12-04T09:43:34.2322763Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2323137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2323473Z res = mod(**inputs) 2025-12-04T09:43:34.2323873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2324329Z outputs = self.mobilebert( 2025-12-04T09:43:34.2324758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2325189Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2325606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2326038Z layer_outputs = layer_module( 2025-12-04T09:43:34.2326459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2326909Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2327354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2327838Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2328330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2328806Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2329290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2329742Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2329898Z 2025-12-04T09:43:34.2330012Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2330377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2330716Z res = mod(**inputs) 2025-12-04T09:43:34.2331127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2331678Z outputs = self.mobilebert( 2025-12-04T09:43:34.2332115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2332568Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2333049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2333485Z layer_outputs = layer_module( 2025-12-04T09:43:34.2333918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2334384Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2334848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2335328Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2335811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2336313Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2336482Z 2025-12-04T09:43:34.2336607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2336993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2337342Z res = mod(**inputs) 2025-12-04T09:43:34.2337765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2338202Z outputs = self.mobilebert( 2025-12-04T09:43:34.2338635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2339084Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2339524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2339988Z layer_outputs = layer_module( 2025-12-04T09:43:34.2340462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2340939Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2341426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2341934Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2342387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2342848Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2343014Z 2025-12-04T09:43:34.2343125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2343484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2343812Z res = mod(**inputs) 2025-12-04T09:43:34.2344208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2344615Z outputs = self.mobilebert( 2025-12-04T09:43:34.2345020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2345434Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2345846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2346328Z layer_outputs = layer_module( 2025-12-04T09:43:34.2346744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2347186Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2347613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2348099Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2348564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2348989Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2349129Z 2025-12-04T09:43:34.2349229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2349583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2349900Z res = mod(**inputs) 2025-12-04T09:43:34.2350284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2350700Z outputs = self.mobilebert( 2025-12-04T09:43:34.2351116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2351531Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2351926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2352344Z layer_outputs = layer_module( 2025-12-04T09:43:34.2352742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2353169Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2353591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2354047Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2354527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2354984Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2355433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2355866Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2356011Z 2025-12-04T09:43:34.2356118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2356471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2356781Z res = mod(**inputs) 2025-12-04T09:43:34.2357165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2357577Z outputs = self.mobilebert( 2025-12-04T09:43:34.2357965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2358379Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2358781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2359194Z layer_outputs = layer_module( 2025-12-04T09:43:34.2359589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2360020Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2360451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2360912Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2361378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2361825Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2361970Z 2025-12-04T09:43:34.2362107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2362480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2362821Z res = mod(**inputs) 2025-12-04T09:43:34.2363215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2363650Z outputs = self.mobilebert( 2025-12-04T09:43:34.2364067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2364503Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2364957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2365390Z layer_outputs = layer_module( 2025-12-04T09:43:34.2365830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2366288Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2366741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2367210Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2367678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2368156Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2368328Z 2025-12-04T09:43:34.2368442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2368836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2369180Z res = mod(**inputs) 2025-12-04T09:43:34.2369587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2370024Z outputs = self.mobilebert( 2025-12-04T09:43:34.2370437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2370878Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2371355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2371772Z layer_outputs = layer_module( 2025-12-04T09:43:34.2372202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2372823Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2373318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2373833Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2374348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2374801Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2374951Z 2025-12-04T09:43:34.2375067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2375434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2375775Z res = mod(**inputs) 2025-12-04T09:43:34.2376189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2376624Z outputs = self.mobilebert( 2025-12-04T09:43:34.2377093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2377541Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2377949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2378349Z layer_outputs = layer_module( 2025-12-04T09:43:34.2378757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2379190Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2379627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2380126Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2380640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2381133Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2381615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2382078Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2382245Z 2025-12-04T09:43:34.2382353Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2382730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2383061Z res = mod(**inputs) 2025-12-04T09:43:34.2383476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2383934Z outputs = self.mobilebert( 2025-12-04T09:43:34.2384360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2384797Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2385223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2385687Z layer_outputs = layer_module( 2025-12-04T09:43:34.2386107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2386595Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2387079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2387525Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2387676Z 2025-12-04T09:43:34.2387784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2388160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2388499Z res = mod(**inputs) 2025-12-04T09:43:34.2388901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2389326Z outputs = self.mobilebert( 2025-12-04T09:43:34.2389751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2390186Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2390604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2391037Z layer_outputs = layer_module( 2025-12-04T09:43:34.2391471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2391958Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2392460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2392934Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2393114Z 2025-12-04T09:43:34.2393222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2393588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2393913Z res = mod(**inputs) 2025-12-04T09:43:34.2394313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2394744Z outputs = self.mobilebert( 2025-12-04T09:43:34.2395183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2395634Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2396069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2396503Z layer_outputs = layer_module( 2025-12-04T09:43:34.2396932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2397466Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2398012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.2398485Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.2398649Z 2025-12-04T09:43:34.2398794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2399170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2399511Z res = mod(**inputs) 2025-12-04T09:43:34.2399910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2400346Z outputs = self.mobilebert( 2025-12-04T09:43:34.2400772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2401231Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2401650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2402083Z layer_outputs = layer_module( 2025-12-04T09:43:34.2402509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2403038Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2403559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.2404041Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.2404523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2404974Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2405130Z 2025-12-04T09:43:34.2405235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2405609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2405945Z res = mod(**inputs) 2025-12-04T09:43:34.2406344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2406788Z outputs = self.mobilebert( 2025-12-04T09:43:34.2407250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2407704Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2408145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2408595Z layer_outputs = layer_module( 2025-12-04T09:43:34.2409033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2409582Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2410127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2410649Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2411165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.2411720Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2411884Z 2025-12-04T09:43:34.2411995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2412381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2412731Z res = mod(**inputs) 2025-12-04T09:43:34.2413143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2413595Z outputs = self.mobilebert( 2025-12-04T09:43:34.2414033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2414510Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2414941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2415381Z layer_outputs = layer_module( 2025-12-04T09:43:34.2415815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2416337Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2416873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2417366Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2417859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.2418357Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2418853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2419321Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2419480Z 2025-12-04T09:43:34.2419598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2419977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2420326Z res = mod(**inputs) 2025-12-04T09:43:34.2420741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2421175Z outputs = self.mobilebert( 2025-12-04T09:43:34.2421603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2422039Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2422483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2422910Z layer_outputs = layer_module( 2025-12-04T09:43:34.2423315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2423486Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2423774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.2423883Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.2424172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.2424277Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.2424281Z 2025-12-04T09:43:34.2425245Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2425482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2425552Z res = mod(**inputs) 2025-12-04T09:43:34.2425851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2425926Z outputs = self.mobilebert( 2025-12-04T09:43:34.2426214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2426297Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2426584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2426688Z layer_outputs = layer_module( 2025-12-04T09:43:34.2426993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2427086Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2427395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2427465Z self_outputs = self.self( 2025-12-04T09:43:34.2427750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T09:43:34.2427830Z self.value(value_tensor) 2025-12-04T09:43:34.2427833Z 2025-12-04T09:43:34.2427937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2428144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2428210Z res = mod(**inputs) 2025-12-04T09:43:34.2428500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2428579Z outputs = self.mobilebert( 2025-12-04T09:43:34.2428866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2428947Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2429232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2429300Z layer_outputs = layer_module( 2025-12-04T09:43:34.2429591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2429750Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2430042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T09:43:34.2430160Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T09:43:34.2430464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T09:43:34.2430555Z layer_input = self.dense(hidden_states) 2025-12-04T09:43:34.2430558Z 2025-12-04T09:43:34.2430663Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2430854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2430926Z res = mod(**inputs) 2025-12-04T09:43:34.2431202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2431280Z outputs = self.mobilebert( 2025-12-04T09:43:34.2431575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2431648Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2431949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2432020Z layer_outputs = layer_module( 2025-12-04T09:43:34.2432297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T09:43:34.2432462Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T09:43:34.2432742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T09:43:34.2432857Z shared_attention_input = self.attention(hidden_states) 2025-12-04T09:43:34.2433134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T09:43:34.2433243Z layer_input = self.LayerNorm(layer_input) 2025-12-04T09:43:34.2433547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2433645Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2433649Z 2025-12-04T09:43:34.2433765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2433974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2434040Z res = mod(**inputs) 2025-12-04T09:43:34.2434341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2434413Z outputs = self.mobilebert( 2025-12-04T09:43:34.2434718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2434797Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2435095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2435175Z layer_outputs = layer_module( 2025-12-04T09:43:34.2435471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2435558Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2435865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2435936Z self_outputs = self.self( 2025-12-04T09:43:34.2436222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T09:43:34.2436295Z self.query(query_tensor) 2025-12-04T09:43:34.2436298Z 2025-12-04T09:43:34.2436400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2436621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2436687Z res = mod(**inputs) 2025-12-04T09:43:34.2436970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2437039Z outputs = self.mobilebert( 2025-12-04T09:43:34.2437316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2437392Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2437669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2437754Z layer_outputs = layer_module( 2025-12-04T09:43:34.2438047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2438147Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2438436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T09:43:34.2438506Z self_outputs = self.self( 2025-12-04T09:43:34.2438784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T09:43:34.2438858Z self.key(key_tensor) 2025-12-04T09:43:34.2438861Z 2025-12-04T09:43:34.2438943Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2439019Z cudagraph partition due to non gpu ops 2025-12-04T09:43:34.2439130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2439326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2439415Z res = mod(**inputs) 2025-12-04T09:43:34.2439699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2439770Z outputs = self.mobilebert( 2025-12-04T09:43:34.2440053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2440124Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2440410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2440479Z layer_outputs = layer_module( 2025-12-04T09:43:34.2440761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2440854Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2441139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2441264Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2441552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T09:43:34.2441636Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2441640Z 2025-12-04T09:43:34.2441748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2441943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2442005Z res = mod(**inputs) 2025-12-04T09:43:34.2442294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2442364Z outputs = self.mobilebert( 2025-12-04T09:43:34.2442650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2442721Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2443029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2443106Z layer_outputs = layer_module( 2025-12-04T09:43:34.2443390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T09:43:34.2443471Z self_attention_outputs = self.attention( 2025-12-04T09:43:34.2443763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T09:43:34.2443884Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T09:43:34.2444201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T09:43:34.2444351Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2444649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2444753Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2444757Z 2025-12-04T09:43:34.2444865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2445078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2445145Z res = mod(**inputs) 2025-12-04T09:43:34.2445445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2445528Z outputs = self.mobilebert( 2025-12-04T09:43:34.2445825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2445926Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2446232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2446306Z layer_outputs = layer_module( 2025-12-04T09:43:34.2446614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2446715Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2447017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2447144Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2447441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2447536Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2447540Z 2025-12-04T09:43:34.2447648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2447857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2447929Z res = mod(**inputs) 2025-12-04T09:43:34.2448228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2448309Z outputs = self.mobilebert( 2025-12-04T09:43:34.2448612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2448685Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2448992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2449067Z layer_outputs = layer_module( 2025-12-04T09:43:34.2449372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2449492Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2449790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2449913Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2450209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2450325Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2450329Z 2025-12-04T09:43:34.2450442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2450648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2450740Z res = mod(**inputs) 2025-12-04T09:43:34.2451060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2451137Z outputs = self.mobilebert( 2025-12-04T09:43:34.2451534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2451617Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2451922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2452008Z layer_outputs = layer_module( 2025-12-04T09:43:34.2452308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2452425Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2452755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2452889Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2453192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2453278Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2453282Z 2025-12-04T09:43:34.2453397Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2453602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2453669Z res = mod(**inputs) 2025-12-04T09:43:34.2453981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2454059Z outputs = self.mobilebert( 2025-12-04T09:43:34.2454367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2454453Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2454759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2454841Z layer_outputs = layer_module( 2025-12-04T09:43:34.2455145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2455244Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2455563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2455697Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2456024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2456157Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2456490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2456599Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2456603Z 2025-12-04T09:43:34.2456713Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2456945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2457012Z res = mod(**inputs) 2025-12-04T09:43:34.2457322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2457403Z outputs = self.mobilebert( 2025-12-04T09:43:34.2457715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2457816Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2458118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2458190Z layer_outputs = layer_module( 2025-12-04T09:43:34.2458490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2458585Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2458883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2459005Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2459304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2459420Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2459426Z 2025-12-04T09:43:34.2459535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2459738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2459813Z res = mod(**inputs) 2025-12-04T09:43:34.2460103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2460176Z outputs = self.mobilebert( 2025-12-04T09:43:34.2460476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2460550Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2460848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2460925Z layer_outputs = layer_module( 2025-12-04T09:43:34.2461220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2461325Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2461622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2461744Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2462038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2462153Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2462157Z 2025-12-04T09:43:34.2462273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2462481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2462555Z res = mod(**inputs) 2025-12-04T09:43:34.2462866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2462943Z outputs = self.mobilebert( 2025-12-04T09:43:34.2463244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2463318Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2463620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2463700Z layer_outputs = layer_module( 2025-12-04T09:43:34.2463990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2464112Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2464428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2464562Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2464866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2464951Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2464955Z 2025-12-04T09:43:34.2465068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2465277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2465343Z res = mod(**inputs) 2025-12-04T09:43:34.2465654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2465751Z outputs = self.mobilebert( 2025-12-04T09:43:34.2466046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2466130Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2466427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2466505Z layer_outputs = layer_module( 2025-12-04T09:43:34.2466803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2466898Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2467202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2467329Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2467636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2467766Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2468061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2468165Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2468169Z 2025-12-04T09:43:34.2468276Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2468487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2468553Z res = mod(**inputs) 2025-12-04T09:43:34.2468851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2468936Z outputs = self.mobilebert( 2025-12-04T09:43:34.2469237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2469313Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2469637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2469713Z layer_outputs = layer_module( 2025-12-04T09:43:34.2470015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2470112Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2470407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2470531Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2470845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2470954Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2470958Z 2025-12-04T09:43:34.2471069Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2471274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2471351Z res = mod(**inputs) 2025-12-04T09:43:34.2471646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2471719Z outputs = self.mobilebert( 2025-12-04T09:43:34.2472024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2472097Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2472570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2472700Z layer_outputs = layer_module( 2025-12-04T09:43:34.2473008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2473113Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2473407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T09:43:34.2473531Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T09:43:34.2473827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2473943Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2473947Z 2025-12-04T09:43:34.2474063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2474270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2474339Z res = mod(**inputs) 2025-12-04T09:43:34.2474644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2474718Z outputs = self.mobilebert( 2025-12-04T09:43:34.2475020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2475093Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2475389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2475472Z layer_outputs = layer_module( 2025-12-04T09:43:34.2475765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2475875Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2476201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2476332Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2476633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T09:43:34.2476719Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2476723Z 2025-12-04T09:43:34.2476837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2477046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2477112Z res = mod(**inputs) 2025-12-04T09:43:34.2477476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2477581Z outputs = self.mobilebert( 2025-12-04T09:43:34.2477903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2477987Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2478283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2478363Z layer_outputs = layer_module( 2025-12-04T09:43:34.2478658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T09:43:34.2478752Z attention_output = ffn_module(attention_output) 2025-12-04T09:43:34.2479055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T09:43:34.2479193Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T09:43:34.2479504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T09:43:34.2479629Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2479915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2480014Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2480017Z 2025-12-04T09:43:34.2480119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2480317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2480388Z res = mod(**inputs) 2025-12-04T09:43:34.2480691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2480775Z outputs = self.mobilebert( 2025-12-04T09:43:34.2481074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2481150Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2481459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2481531Z layer_outputs = layer_module( 2025-12-04T09:43:34.2481838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2481965Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2482262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T09:43:34.2482357Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2482364Z 2025-12-04T09:43:34.2482472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2482682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2482778Z res = mod(**inputs) 2025-12-04T09:43:34.2483075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2483153Z outputs = self.mobilebert( 2025-12-04T09:43:34.2483456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2483529Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2483829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2483899Z layer_outputs = layer_module( 2025-12-04T09:43:34.2484227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T09:43:34.2484370Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:43:34.2484666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T09:43:34.2484788Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:43:34.2484791Z 2025-12-04T09:43:34.2484898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2485104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2485178Z res = mod(**inputs) 2025-12-04T09:43:34.2485470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2485549Z outputs = self.mobilebert( 2025-12-04T09:43:34.2485844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2485963Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2486268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2486340Z layer_outputs = layer_module( 2025-12-04T09:43:34.2486643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2486810Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2487104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T09:43:34.2487208Z layer_output = self.dense(intermediate_states) 2025-12-04T09:43:34.2487212Z 2025-12-04T09:43:34.2487320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2487534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2487605Z res = mod(**inputs) 2025-12-04T09:43:34.2487900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2487980Z outputs = self.mobilebert( 2025-12-04T09:43:34.2488276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2488350Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2488650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2488722Z layer_outputs = layer_module( 2025-12-04T09:43:34.2489026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2489194Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2489505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T09:43:34.2489643Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T09:43:34.2489938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2490043Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2490047Z 2025-12-04T09:43:34.2490155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2490359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2490436Z res = mod(**inputs) 2025-12-04T09:43:34.2490746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2490821Z outputs = self.mobilebert( 2025-12-04T09:43:34.2491140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2491274Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2491598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2491674Z layer_outputs = layer_module( 2025-12-04T09:43:34.2491978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2492154Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2492458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2492621Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2492928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T09:43:34.2493019Z layer_outputs = self.dense(hidden_states) 2025-12-04T09:43:34.2493023Z 2025-12-04T09:43:34.2493141Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2493352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2493430Z res = mod(**inputs) 2025-12-04T09:43:34.2493749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T09:43:34.2493824Z outputs = self.mobilebert( 2025-12-04T09:43:34.2494131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T09:43:34.2494210Z encoder_outputs = self.encoder( 2025-12-04T09:43:34.2494518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T09:43:34.2494600Z layer_outputs = layer_module( 2025-12-04T09:43:34.2494896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T09:43:34.2495067Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T09:43:34.2495364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T09:43:34.2495490Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T09:43:34.2495790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T09:43:34.2495918Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T09:43:34.2496240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T09:43:34.2496338Z return input_tensor * self.weight + self.bias 2025-12-04T09:43:34.2496342Z 2025-12-04T09:43:34.2496448Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2496661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2496727Z res = mod(**inputs) 2025-12-04T09:43:34.2497022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T09:43:34.2497129Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:43:34.2497425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T09:43:34.2497572Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:43:34.2497889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-12-04T09:43:34.2497986Z hidden_states = self.transform(hidden_states) 2025-12-04T09:43:34.2498287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-12-04T09:43:34.2498371Z hidden_states = self.dense(hidden_states) 2025-12-04T09:43:34.2498375Z 2025-12-04T09:43:34.2498487Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2498692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2498758Z res = mod(**inputs) 2025-12-04T09:43:34.2499058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T09:43:34.2499173Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:43:34.2499479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T09:43:34.2499595Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:43:34.2499887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-12-04T09:43:34.2500112Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-12-04T09:43:34.2500116Z 2025-12-04T09:43:34.2500221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2500432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2500496Z res = mod(**inputs) 2025-12-04T09:43:34.2500796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T09:43:34.2500896Z prediction_scores = self.cls(sequence_output) 2025-12-04T09:43:34.2501192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T09:43:34.2501302Z prediction_scores = self.predictions(sequence_output) 2025-12-04T09:43:34.2501602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-12-04T09:43:34.2501683Z hidden_states += self.decoder.bias 2025-12-04T09:43:34.2501687Z 2025-12-04T09:43:34.2501797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:43:34.2502000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:43:34.2502064Z res = mod(**inputs) 2025-12-04T09:43:34.2502365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-12-04T09:43:34.2502580Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:43:34.2502584Z 2025-12-04T09:43:47.4801623Z Compilation time (from dynamo_timed): 38.757548797 2025-12-04T09:43:47.4801977Z pass 2025-12-04T09:43:47.4802323Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:47.4803203Z TIMING: _recursive_pre_grad_passes:0.02264 _recursive_joint_graph_passes:1.31371 _recursive_post_grad_passes:0.19249 async_compile.wait:0.79144 code_gen:11.60398 inductor_compile:16.02084 backend_compile:27.67684 gc:0.00049 entire_frame_compile:38.75755 total_wall_time:38.75755 2025-12-04T09:43:47.4804217Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:30412 | FakeTensor.__torch_dispatch__:15340 | ProxyTorchDispatchMode.__torch_dispatch__:8821 2025-12-04T09:43:47.4804993Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-12-04T09:43:50.9928535Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:43:50.9929375Z import pynvml # type: ignore[import] 2025-12-04T09:43:54.3897188Z 2025-12-04T09:43:55.9313354Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:43:55.9313796Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:43:55.9329797Z cpu eval OPTForCausalLM 2025-12-04T09:43:57.8192985Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:58.6310796Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:43:59.4616900Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:06.4072645Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4073040Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4073263Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4073470Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4073682Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4073880Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4074084Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4074286Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4074483Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4074690Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4074973Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4075178Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4075413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4075817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4076161Z res = mod(**inputs) 2025-12-04T09:44:06.4076533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4076931Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4077359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4077785Z outputs = self.model.decoder( 2025-12-04T09:44:06.4078154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4078534Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4078943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4079348Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4079734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4080141Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4080872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4081290Z return func(*args, **kwargs) 2025-12-04T09:44:06.4081693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4082138Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4082571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4083013Z return func(*args, **kwargs) 2025-12-04T09:44:06.4083413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4083950Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4084129Z 2025-12-04T09:44:06.4084295Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4084702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4085075Z res = mod(**inputs) 2025-12-04T09:44:06.4085432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4085828Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4086249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4086676Z outputs = self.model.decoder( 2025-12-04T09:44:06.4087057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4087473Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4087933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4088347Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4088721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4089132Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4089560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4089969Z return func(*args, **kwargs) 2025-12-04T09:44:06.4090371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4090820Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4091408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4091843Z return func(*args, **kwargs) 2025-12-04T09:44:06.4092259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4092715Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4092863Z 2025-12-04T09:44:06.4092986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4093379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4093730Z res = mod(**inputs) 2025-12-04T09:44:06.4094084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4094467Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4094882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4095293Z outputs = self.model.decoder( 2025-12-04T09:44:06.4095667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4096054Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4096484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4096897Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4097270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4097660Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4098062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4098473Z return func(*args, **kwargs) 2025-12-04T09:44:06.4098849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4099302Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4099749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4100140Z return func(*args, **kwargs) 2025-12-04T09:44:06.4100522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4100922Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4101061Z 2025-12-04T09:44:06.4101149Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4101354Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4101590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4101957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4102289Z res = mod(**inputs) 2025-12-04T09:44:06.4102633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4103040Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4103438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4103833Z outputs = self.model.decoder( 2025-12-04T09:44:06.4104183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4104543Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4104917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4105290Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4105647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4106030Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4106427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4106809Z return func(*args, **kwargs) 2025-12-04T09:44:06.4107172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4107581Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4107992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4108381Z return func(*args, **kwargs) 2025-12-04T09:44:06.4108763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4109178Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4109655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4110174Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4110368Z 2025-12-04T09:44:06.4110689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4111086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4111434Z res = mod(**inputs) 2025-12-04T09:44:06.4111781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4112168Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4112562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4112965Z outputs = self.model.decoder( 2025-12-04T09:44:06.4113331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4113723Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4114124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4114590Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4114974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4115360Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4115770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4116181Z return func(*args, **kwargs) 2025-12-04T09:44:06.4116574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4117006Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4117432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4117860Z return func(*args, **kwargs) 2025-12-04T09:44:06.4118243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4118671Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4118824Z 2025-12-04T09:44:06.4118936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4119317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4119655Z res = mod(**inputs) 2025-12-04T09:44:06.4120002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4120396Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4120805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4121213Z outputs = self.model.decoder( 2025-12-04T09:44:06.4121573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4121958Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4122347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4122744Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4123114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4123496Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4123896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4124313Z return func(*args, **kwargs) 2025-12-04T09:44:06.4124694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4125097Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4125249Z 2025-12-04T09:44:06.4125361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4125761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4126103Z res = mod(**inputs) 2025-12-04T09:44:06.4126441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4126816Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4127209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4127599Z outputs = self.model.decoder( 2025-12-04T09:44:06.4127963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4128358Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4128749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4129158Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4129546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4129941Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4130354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4130751Z return func(*args, **kwargs) 2025-12-04T09:44:06.4131146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4131725Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4131926Z 2025-12-04T09:44:06.4132040Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4133275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4133619Z res = mod(**inputs) 2025-12-04T09:44:06.4133974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4134352Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4134756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4135162Z outputs = self.model.decoder( 2025-12-04T09:44:06.4135525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4135908Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4136308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4136713Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4137088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4137480Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4137897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4138294Z return func(*args, **kwargs) 2025-12-04T09:44:06.4138686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4139114Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4139262Z 2025-12-04T09:44:06.4139381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4139756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4140102Z res = mod(**inputs) 2025-12-04T09:44:06.4140454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4140839Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4141258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4141670Z outputs = self.model.decoder( 2025-12-04T09:44:06.4142045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4142427Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4142831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4143246Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4143625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4144020Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4144461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4144866Z return func(*args, **kwargs) 2025-12-04T09:44:06.4145225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4145627Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4146025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4146398Z return func(*args, **kwargs) 2025-12-04T09:44:06.4146751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4147173Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4147339Z 2025-12-04T09:44:06.4147452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4147831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4148152Z res = mod(**inputs) 2025-12-04T09:44:06.4148479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4148835Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4149205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4149583Z outputs = self.model.decoder( 2025-12-04T09:44:06.4149931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4150285Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4150651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4151024Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4151375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4151730Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4152113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4152483Z return func(*args, **kwargs) 2025-12-04T09:44:06.4152844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4153235Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4153637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4154008Z return func(*args, **kwargs) 2025-12-04T09:44:06.4154363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4154750Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4154893Z 2025-12-04T09:44:06.4154996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4155376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4155695Z res = mod(**inputs) 2025-12-04T09:44:06.4156024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4156381Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4156755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4157124Z outputs = self.model.decoder( 2025-12-04T09:44:06.4157465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4157817Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4158204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4158598Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4158950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4159311Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4159678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4160048Z return func(*args, **kwargs) 2025-12-04T09:44:06.4160407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4160798Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4161203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4161599Z return func(*args, **kwargs) 2025-12-04T09:44:06.4161965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4162351Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4162498Z 2025-12-04T09:44:06.4162580Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4162799Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4163031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4163390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4163712Z res = mod(**inputs) 2025-12-04T09:44:06.4164041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4164395Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4164771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4165153Z outputs = self.model.decoder( 2025-12-04T09:44:06.4165494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4165849Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4166227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4166606Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4166947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4167312Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4167690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4168061Z return func(*args, **kwargs) 2025-12-04T09:44:06.4168447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4168879Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4169371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4169768Z return func(*args, **kwargs) 2025-12-04T09:44:06.4170158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4170589Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4171066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4171671Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4171880Z 2025-12-04T09:44:06.4171992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4172575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4172932Z res = mod(**inputs) 2025-12-04T09:44:06.4173346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4173735Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4174146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4174563Z outputs = self.model.decoder( 2025-12-04T09:44:06.4174946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4175337Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4175749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4176163Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4176561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4176946Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4177340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4177729Z return func(*args, **kwargs) 2025-12-04T09:44:06.4178110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4178537Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4178946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4179344Z return func(*args, **kwargs) 2025-12-04T09:44:06.4179726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4180131Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4180275Z 2025-12-04T09:44:06.4180384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4180760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4181099Z res = mod(**inputs) 2025-12-04T09:44:06.4181429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4181809Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4182201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4182607Z outputs = self.model.decoder( 2025-12-04T09:44:06.4182961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4183335Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4183731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4184131Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4184528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4184916Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4185315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4185703Z return func(*args, **kwargs) 2025-12-04T09:44:06.4186087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4186500Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4186643Z 2025-12-04T09:44:06.4186753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4187171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4187514Z res = mod(**inputs) 2025-12-04T09:44:06.4187882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4188228Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4188600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4188976Z outputs = self.model.decoder( 2025-12-04T09:44:06.4189318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4189662Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4190031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4190405Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4190768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4191135Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4191514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4191882Z return func(*args, **kwargs) 2025-12-04T09:44:06.4192234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4192633Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4192783Z 2025-12-04T09:44:06.4192893Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4193242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4193558Z res = mod(**inputs) 2025-12-04T09:44:06.4193880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4194231Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4194597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4194972Z outputs = self.model.decoder( 2025-12-04T09:44:06.4195310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4195650Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4196018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4196393Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4196739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4197094Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4197469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4197838Z return func(*args, **kwargs) 2025-12-04T09:44:06.4198221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4198600Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4198743Z 2025-12-04T09:44:06.4198850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4199213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4199533Z res = mod(**inputs) 2025-12-04T09:44:06.4199872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4200234Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4200614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4201008Z outputs = self.model.decoder( 2025-12-04T09:44:06.4201373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4201727Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4202090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4202496Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4202848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4203212Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4203579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4203950Z return func(*args, **kwargs) 2025-12-04T09:44:06.4204331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4204735Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4205127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4205514Z return func(*args, **kwargs) 2025-12-04T09:44:06.4205893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4206325Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4206506Z 2025-12-04T09:44:06.4206616Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4206994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4207331Z res = mod(**inputs) 2025-12-04T09:44:06.4207666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4208046Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4208443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4208828Z outputs = self.model.decoder( 2025-12-04T09:44:06.4209191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4209559Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4209951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4210336Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4210702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4211082Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4211568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4211971Z return func(*args, **kwargs) 2025-12-04T09:44:06.4212376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4212804Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4213218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4213607Z return func(*args, **kwargs) 2025-12-04T09:44:06.4213990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4214394Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4214538Z 2025-12-04T09:44:06.4214648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4215049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4215387Z res = mod(**inputs) 2025-12-04T09:44:06.4215737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4216111Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4216504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4216898Z outputs = self.model.decoder( 2025-12-04T09:44:06.4217251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4217618Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4218014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4218399Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4218791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4219174Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4219571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4219954Z return func(*args, **kwargs) 2025-12-04T09:44:06.4220337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4220769Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4221188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4221545Z return func(*args, **kwargs) 2025-12-04T09:44:06.4221904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4222287Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4222426Z 2025-12-04T09:44:06.4222507Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4222722Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4222960Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4223317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4223630Z res = mod(**inputs) 2025-12-04T09:44:06.4223952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4224304Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4224671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4225046Z outputs = self.model.decoder( 2025-12-04T09:44:06.4225390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4225740Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4226102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4226506Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4226863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4227223Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4227602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4227975Z return func(*args, **kwargs) 2025-12-04T09:44:06.4228339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4228737Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4229155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4229526Z return func(*args, **kwargs) 2025-12-04T09:44:06.4229894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4230298Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4230738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4231216Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4231398Z 2025-12-04T09:44:06.4231502Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4231859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4232180Z res = mod(**inputs) 2025-12-04T09:44:06.4232504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4232875Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4233457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4233846Z outputs = self.model.decoder( 2025-12-04T09:44:06.4234189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4234546Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4234924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4235301Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4235645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4236016Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4236405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4236789Z return func(*args, **kwargs) 2025-12-04T09:44:06.4237146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4237550Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4237949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4238313Z return func(*args, **kwargs) 2025-12-04T09:44:06.4238675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4239065Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4239201Z 2025-12-04T09:44:06.4239311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4239663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4239983Z res = mod(**inputs) 2025-12-04T09:44:06.4240333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4240689Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4241053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4241419Z outputs = self.model.decoder( 2025-12-04T09:44:06.4241751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4242092Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4242450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4242841Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4243177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4243549Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4243921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4244284Z return func(*args, **kwargs) 2025-12-04T09:44:06.4244639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4245019Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4245156Z 2025-12-04T09:44:06.4245267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4245626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4245941Z res = mod(**inputs) 2025-12-04T09:44:06.4246260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4246623Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4246981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4247353Z outputs = self.model.decoder( 2025-12-04T09:44:06.4247698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4248045Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4248409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4248782Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4249133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4249486Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4249867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4250241Z return func(*args, **kwargs) 2025-12-04T09:44:06.4250605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4250996Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4251162Z 2025-12-04T09:44:06.4251326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4251717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4252053Z res = mod(**inputs) 2025-12-04T09:44:06.4252388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4252758Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4253126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4253499Z outputs = self.model.decoder( 2025-12-04T09:44:06.4253868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4254216Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4254585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4254953Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4255306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4255681Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4256065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4256431Z return func(*args, **kwargs) 2025-12-04T09:44:06.4256807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4257224Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4257359Z 2025-12-04T09:44:06.4257461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4257808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4258126Z res = mod(**inputs) 2025-12-04T09:44:06.4258448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4258792Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4259162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4259539Z outputs = self.model.decoder( 2025-12-04T09:44:06.4259872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4260255Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4260627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4261003Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4261346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4261712Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4262087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4262453Z return func(*args, **kwargs) 2025-12-04T09:44:06.4262814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:44:06.4263244Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:44:06.4263435Z 2025-12-04T09:44:06.4263546Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4263900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4264222Z res = mod(**inputs) 2025-12-04T09:44:06.4264544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4264896Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4265269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4265647Z outputs = self.model.decoder( 2025-12-04T09:44:06.4266000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4266351Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4266722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4267101Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4267442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4267824Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4268206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4268577Z return func(*args, **kwargs) 2025-12-04T09:44:06.4268934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4269341Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4269744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4270123Z return func(*args, **kwargs) 2025-12-04T09:44:06.4270501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4270934Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4271105Z 2025-12-04T09:44:06.4271221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4271569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4271904Z res = mod(**inputs) 2025-12-04T09:44:06.4272222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4272690Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4273051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4273423Z outputs = self.model.decoder( 2025-12-04T09:44:06.4273761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4274162Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4274536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4274910Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4275262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4275607Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4275978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4276338Z return func(*args, **kwargs) 2025-12-04T09:44:06.4276689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4277072Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4277468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4277828Z return func(*args, **kwargs) 2025-12-04T09:44:06.4278173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4278544Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4278680Z 2025-12-04T09:44:06.4278779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4279127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4279438Z res = mod(**inputs) 2025-12-04T09:44:06.4279758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4280106Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4280469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4280843Z outputs = self.model.decoder( 2025-12-04T09:44:06.4281213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4281565Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4281931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4282307Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4282656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4283017Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4283387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4283758Z return func(*args, **kwargs) 2025-12-04T09:44:06.4284154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4284610Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4285019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4285388Z return func(*args, **kwargs) 2025-12-04T09:44:06.4285749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4286126Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4286272Z 2025-12-04T09:44:06.4286353Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4286562Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4286789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4287146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4287491Z res = mod(**inputs) 2025-12-04T09:44:06.4287821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4288173Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4288551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4288932Z outputs = self.model.decoder( 2025-12-04T09:44:06.4289288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4289664Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4290060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4290458Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4290823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4291280Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4291704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4292114Z return func(*args, **kwargs) 2025-12-04T09:44:06.4292512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4292950Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4293376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4293766Z return func(*args, **kwargs) 2025-12-04T09:44:06.4294199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4294615Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4295066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4295562Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4295777Z 2025-12-04T09:44:06.4295883Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4296243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4296560Z res = mod(**inputs) 2025-12-04T09:44:06.4296882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4297235Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4297613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4297984Z outputs = self.model.decoder( 2025-12-04T09:44:06.4298346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4298701Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4299093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4299471Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4299818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4300177Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4300544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4300917Z return func(*args, **kwargs) 2025-12-04T09:44:06.4301276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4301676Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4302090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4302462Z return func(*args, **kwargs) 2025-12-04T09:44:06.4302822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4303199Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4303343Z 2025-12-04T09:44:06.4303446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4303805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4304126Z res = mod(**inputs) 2025-12-04T09:44:06.4304443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4304798Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4305172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4305541Z outputs = self.model.decoder( 2025-12-04T09:44:06.4305875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4306215Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4306574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4306929Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4307269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4307623Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4307993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4308349Z return func(*args, **kwargs) 2025-12-04T09:44:06.4308702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4309073Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4309205Z 2025-12-04T09:44:06.4309330Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4309680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4309993Z res = mod(**inputs) 2025-12-04T09:44:06.4310303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4310635Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4310994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4311356Z outputs = self.model.decoder( 2025-12-04T09:44:06.4311676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4312037Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4312417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4312783Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4313121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4313481Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4313848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4314206Z return func(*args, **kwargs) 2025-12-04T09:44:06.4314560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4314949Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4315118Z 2025-12-04T09:44:06.4315225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4315568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4315883Z res = mod(**inputs) 2025-12-04T09:44:06.4316201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4316544Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4316907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4317276Z outputs = self.model.decoder( 2025-12-04T09:44:06.4317612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4317948Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4318313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4318682Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4319030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4319379Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4319749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4320112Z return func(*args, **kwargs) 2025-12-04T09:44:06.4320470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4320834Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4320969Z 2025-12-04T09:44:06.4321067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4321405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4321708Z res = mod(**inputs) 2025-12-04T09:44:06.4322020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4322357Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4322723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4323081Z outputs = self.model.decoder( 2025-12-04T09:44:06.4323406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4323743Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4324090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4324449Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4324803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4325189Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4325590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4325966Z return func(*args, **kwargs) 2025-12-04T09:44:06.4326327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4326721Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4327125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4327491Z return func(*args, **kwargs) 2025-12-04T09:44:06.4327848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4328251Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4328448Z 2025-12-04T09:44:06.4328551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4328913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4329227Z res = mod(**inputs) 2025-12-04T09:44:06.4329552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4329906Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4330279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4330651Z outputs = self.model.decoder( 2025-12-04T09:44:06.4330996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4331446Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4331843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4332241Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4332617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4333010Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4333407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4333809Z return func(*args, **kwargs) 2025-12-04T09:44:06.4334193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4334622Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4335039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4335440Z return func(*args, **kwargs) 2025-12-04T09:44:06.4335831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4336233Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4336386Z 2025-12-04T09:44:06.4336523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4336912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4337259Z res = mod(**inputs) 2025-12-04T09:44:06.4337602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4337985Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4338394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4338812Z outputs = self.model.decoder( 2025-12-04T09:44:06.4339179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4339583Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4340001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4340404Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4340776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4341167Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4341579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4341960Z return func(*args, **kwargs) 2025-12-04T09:44:06.4342341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4342766Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4343203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4343604Z return func(*args, **kwargs) 2025-12-04T09:44:06.4343987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4344392Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4344543Z 2025-12-04T09:44:06.4344641Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4344934Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4345185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4345555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4345873Z res = mod(**inputs) 2025-12-04T09:44:06.4346198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4346556Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4346921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4347307Z outputs = self.model.decoder( 2025-12-04T09:44:06.4347673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4348041Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4348434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4348833Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4349203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4349582Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4349982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4350380Z return func(*args, **kwargs) 2025-12-04T09:44:06.4350766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4351214Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4351642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4352037Z return func(*args, **kwargs) 2025-12-04T09:44:06.4352415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4352839Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4353317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4353830Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4354058Z 2025-12-04T09:44:06.4354164Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4354542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4354875Z res = mod(**inputs) 2025-12-04T09:44:06.4355195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4355555Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4355932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4356310Z outputs = self.model.decoder( 2025-12-04T09:44:06.4356648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4357003Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4357404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4357853Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4358218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4358600Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4359001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4359400Z return func(*args, **kwargs) 2025-12-04T09:44:06.4359774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4360175Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4360574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4360940Z return func(*args, **kwargs) 2025-12-04T09:44:06.4361314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4361705Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4361841Z 2025-12-04T09:44:06.4361953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4362301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4362626Z res = mod(**inputs) 2025-12-04T09:44:06.4362967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4363336Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4363712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4364086Z outputs = self.model.decoder( 2025-12-04T09:44:06.4364441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4364809Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4365230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4365627Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4365988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4366368Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4366766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4367156Z return func(*args, **kwargs) 2025-12-04T09:44:06.4367527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4367933Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4368096Z 2025-12-04T09:44:06.4368213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4368603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4368950Z res = mod(**inputs) 2025-12-04T09:44:06.4369295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4369680Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4370069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4370468Z outputs = self.model.decoder( 2025-12-04T09:44:06.4370834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4371300Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4371718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4372161Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4372737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4373121Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4373534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4373943Z return func(*args, **kwargs) 2025-12-04T09:44:06.4374329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4374756Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4374925Z 2025-12-04T09:44:06.4375034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4375416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4375757Z res = mod(**inputs) 2025-12-04T09:44:06.4376104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4376486Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4376931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4377327Z outputs = self.model.decoder( 2025-12-04T09:44:06.4377696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4378066Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4378452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4378852Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4379222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4379611Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4380008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4380455Z return func(*args, **kwargs) 2025-12-04T09:44:06.4380852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4381268Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4381415Z 2025-12-04T09:44:06.4381527Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4381932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4382262Z res = mod(**inputs) 2025-12-04T09:44:06.4382588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4382990Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4383363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4383767Z outputs = self.model.decoder( 2025-12-04T09:44:06.4384108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4384459Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4384829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4385199Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4385549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4385912Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4386290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4386692Z return func(*args, **kwargs) 2025-12-04T09:44:06.4387061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:44:06.4387504Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:44:06.4387691Z 2025-12-04T09:44:06.4387802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4388155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4388485Z res = mod(**inputs) 2025-12-04T09:44:06.4388812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4389164Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4389538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4389923Z outputs = self.model.decoder( 2025-12-04T09:44:06.4390270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4390624Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4391000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4391375Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4391724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4392089Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4392470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4392844Z return func(*args, **kwargs) 2025-12-04T09:44:06.4393202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4393611Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4394074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4394442Z return func(*args, **kwargs) 2025-12-04T09:44:06.4394815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4395227Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4395390Z 2025-12-04T09:44:06.4395501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4395852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4396174Z res = mod(**inputs) 2025-12-04T09:44:06.4396496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4396877Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4397278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4397686Z outputs = self.model.decoder( 2025-12-04T09:44:06.4398044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4398389Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4398765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4399138Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4399485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4399843Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4400218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4400612Z return func(*args, **kwargs) 2025-12-04T09:44:06.4400976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4401394Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4401802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4402180Z return func(*args, **kwargs) 2025-12-04T09:44:06.4402540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4402931Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4403070Z 2025-12-04T09:44:06.4403185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4403547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4403870Z res = mod(**inputs) 2025-12-04T09:44:06.4404203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4404568Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4404943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4405335Z outputs = self.model.decoder( 2025-12-04T09:44:06.4405704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4406084Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4406475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4406879Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4407254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4407645Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4408043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4408468Z return func(*args, **kwargs) 2025-12-04T09:44:06.4408859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4409279Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4409712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4410123Z return func(*args, **kwargs) 2025-12-04T09:44:06.4410494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4410918Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4411094Z 2025-12-04T09:44:06.4411250Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4411496Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4411774Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4412163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4412509Z res = mod(**inputs) 2025-12-04T09:44:06.4412859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4413239Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4413636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4414044Z outputs = self.model.decoder( 2025-12-04T09:44:06.4414402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4414777Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4415207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4415610Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4415975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4416359Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4416760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4417150Z return func(*args, **kwargs) 2025-12-04T09:44:06.4417543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4417975Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4418393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4418790Z return func(*args, **kwargs) 2025-12-04T09:44:06.4419179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4419598Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4420035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4420515Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4420705Z 2025-12-04T09:44:06.4420810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4421170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4421484Z res = mod(**inputs) 2025-12-04T09:44:06.4421809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4422171Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4422547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4422943Z outputs = self.model.decoder( 2025-12-04T09:44:06.4423296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4423697Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4424072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4424452Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4424805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4425170Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4425548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4425951Z return func(*args, **kwargs) 2025-12-04T09:44:06.4426335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4426729Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4427124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4427497Z return func(*args, **kwargs) 2025-12-04T09:44:06.4427853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4428228Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4428370Z 2025-12-04T09:44:06.4428473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4428827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4429170Z res = mod(**inputs) 2025-12-04T09:44:06.4429489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4429831Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4430185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4430536Z outputs = self.model.decoder( 2025-12-04T09:44:06.4430861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4431195Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4431547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4431895Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4432226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4432573Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4432927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4433280Z return func(*args, **kwargs) 2025-12-04T09:44:06.4433622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4433989Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4434119Z 2025-12-04T09:44:06.4434218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4434557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4434857Z res = mod(**inputs) 2025-12-04T09:44:06.4435156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4435491Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4435849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4436238Z outputs = self.model.decoder( 2025-12-04T09:44:06.4436565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4436903Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4437261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4437627Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4437963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4438318Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4438687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4439066Z return func(*args, **kwargs) 2025-12-04T09:44:06.4439440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4439828Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4439981Z 2025-12-04T09:44:06.4440088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4440430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4440745Z res = mod(**inputs) 2025-12-04T09:44:06.4441060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4441395Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4441764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4442147Z outputs = self.model.decoder( 2025-12-04T09:44:06.4442483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4442820Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4443234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4443589Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4443913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4444265Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4444631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4444995Z return func(*args, **kwargs) 2025-12-04T09:44:06.4445338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4445716Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4445848Z 2025-12-04T09:44:06.4445958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4446305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4446607Z res = mod(**inputs) 2025-12-04T09:44:06.4446919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4447266Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4447617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4447980Z outputs = self.model.decoder( 2025-12-04T09:44:06.4448315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4448661Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4449013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4449381Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4449741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4450090Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4450462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4450832Z return func(*args, **kwargs) 2025-12-04T09:44:06.4451290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4451722Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4452147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4452562Z return func(*args, **kwargs) 2025-12-04T09:44:06.4452826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4452952Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4452957Z 2025-12-04T09:44:06.4453062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4453263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4453334Z res = mod(**inputs) 2025-12-04T09:44:06.4453568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4453641Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4453888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4453983Z outputs = self.model.decoder( 2025-12-04T09:44:06.4454200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4454272Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4454508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4454585Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4454802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4454884Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4455119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4455186Z return func(*args, **kwargs) 2025-12-04T09:44:06.4455428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4455529Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4455764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4455838Z return func(*args, **kwargs) 2025-12-04T09:44:06.4456072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4456159Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4456162Z 2025-12-04T09:44:06.4456263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4456455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4456524Z res = mod(**inputs) 2025-12-04T09:44:06.4456734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4456808Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4457051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4457123Z outputs = self.model.decoder( 2025-12-04T09:44:06.4457368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4457442Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4457674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4457752Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4457970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4458052Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4458287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4458372Z return func(*args, **kwargs) 2025-12-04T09:44:06.4458634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4458733Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4458963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4459036Z return func(*args, **kwargs) 2025-12-04T09:44:06.4459270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4459358Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4459362Z 2025-12-04T09:44:06.4459438Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4459513Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4459620Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4459834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4459895Z res = mod(**inputs) 2025-12-04T09:44:06.4460118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4460188Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4460430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4460501Z outputs = self.model.decoder( 2025-12-04T09:44:06.4460710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4460788Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4461016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4461093Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4461309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4461385Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4461626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4461692Z return func(*args, **kwargs) 2025-12-04T09:44:06.4461923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4462025Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4462259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4462331Z return func(*args, **kwargs) 2025-12-04T09:44:06.4462562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4462662Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4462971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4463105Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4463108Z 2025-12-04T09:44:06.4463213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4463418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4463477Z res = mod(**inputs) 2025-12-04T09:44:06.4463688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4463756Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4463983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4464083Z outputs = self.model.decoder( 2025-12-04T09:44:06.4464312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4464392Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4464631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4464701Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4464924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4464999Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4465239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4465310Z return func(*args, **kwargs) 2025-12-04T09:44:06.4465535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4465668Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4465902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4465971Z return func(*args, **kwargs) 2025-12-04T09:44:06.4466211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4466293Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4466296Z 2025-12-04T09:44:06.4466401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4466590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4466653Z res = mod(**inputs) 2025-12-04T09:44:06.4466865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4466941Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4467168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4467252Z outputs = self.model.decoder( 2025-12-04T09:44:06.4467461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4467542Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4467776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4467848Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4468072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4468151Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4468385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4468463Z return func(*args, **kwargs) 2025-12-04T09:44:06.4468728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4468815Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4468819Z 2025-12-04T09:44:06.4468916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4469101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4469168Z res = mod(**inputs) 2025-12-04T09:44:06.4469373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4469447Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4469673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4469779Z outputs = self.model.decoder( 2025-12-04T09:44:06.4469989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4470075Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4470304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4470379Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4470587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4470667Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4470894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4470956Z return func(*args, **kwargs) 2025-12-04T09:44:06.4471189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4471299Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4471303Z 2025-12-04T09:44:06.4471399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4471591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4471652Z res = mod(**inputs) 2025-12-04T09:44:06.4471862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4471930Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4472158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4472236Z outputs = self.model.decoder( 2025-12-04T09:44:06.4472584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4472685Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4472919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4472993Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4473219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4473298Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4473530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4473609Z return func(*args, **kwargs) 2025-12-04T09:44:06.4473842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4473929Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4473933Z 2025-12-04T09:44:06.4474034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4474229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4474302Z res = mod(**inputs) 2025-12-04T09:44:06.4474569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4474639Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4474873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4474941Z outputs = self.model.decoder( 2025-12-04T09:44:06.4475153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4475221Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4475455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4475598Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4475823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4476117Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4476357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4476426Z return func(*args, **kwargs) 2025-12-04T09:44:06.4476670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:44:06.4476804Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:44:06.4476807Z 2025-12-04T09:44:06.4476912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4477103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4477165Z res = mod(**inputs) 2025-12-04T09:44:06.4477414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4477484Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4477730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4477805Z outputs = self.model.decoder( 2025-12-04T09:44:06.4478008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4478081Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4478311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4478377Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4478593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4478668Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4478896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4478966Z return func(*args, **kwargs) 2025-12-04T09:44:06.4479195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4479296Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4479529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4479595Z return func(*args, **kwargs) 2025-12-04T09:44:06.4479833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4479940Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4479943Z 2025-12-04T09:44:06.4480047Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4480239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4480301Z res = mod(**inputs) 2025-12-04T09:44:06.4480536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4480609Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4480848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4480927Z outputs = self.model.decoder( 2025-12-04T09:44:06.4481152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4481230Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4481461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4481554Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4481774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4481874Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4482109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4482182Z return func(*args, **kwargs) 2025-12-04T09:44:06.4482418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4482520Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4482755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4482821Z return func(*args, **kwargs) 2025-12-04T09:44:06.4483061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4483166Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4483170Z 2025-12-04T09:44:06.4483277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4483470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4483531Z res = mod(**inputs) 2025-12-04T09:44:06.4483747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4483816Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4484051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4484131Z outputs = self.model.decoder( 2025-12-04T09:44:06.4484340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4484418Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4484663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4484734Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4484957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4485031Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4485265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4485341Z return func(*args, **kwargs) 2025-12-04T09:44:06.4485572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4485674Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4485909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4485976Z return func(*args, **kwargs) 2025-12-04T09:44:06.4486215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4486319Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4486323Z 2025-12-04T09:44:06.4486413Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4486493Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4486593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4486798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4486860Z res = mod(**inputs) 2025-12-04T09:44:06.4487078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4487158Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4487417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4487498Z outputs = self.model.decoder( 2025-12-04T09:44:06.4487753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4487827Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4488077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4488147Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4488363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4488445Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4488679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4488753Z return func(*args, **kwargs) 2025-12-04T09:44:06.4489003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4489101Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4489342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4489408Z return func(*args, **kwargs) 2025-12-04T09:44:06.4489638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4489742Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4490036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4490172Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4490176Z 2025-12-04T09:44:06.4490278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4490469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4490541Z res = mod(**inputs) 2025-12-04T09:44:06.4490750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4490828Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4491077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4491153Z outputs = self.model.decoder( 2025-12-04T09:44:06.4491485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4491565Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4491820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4491907Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4492143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4492261Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4492517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4492589Z return func(*args, **kwargs) 2025-12-04T09:44:06.4492860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4492955Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4493204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4493273Z return func(*args, **kwargs) 2025-12-04T09:44:06.4493516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4493627Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4493630Z 2025-12-04T09:44:06.4493747Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4493939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4494012Z res = mod(**inputs) 2025-12-04T09:44:06.4494221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4494297Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4494525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4494596Z outputs = self.model.decoder( 2025-12-04T09:44:06.4494810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4494901Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4495135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4495212Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4495428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4495514Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4495748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4495813Z return func(*args, **kwargs) 2025-12-04T09:44:06.4496052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4496131Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4496135Z 2025-12-04T09:44:06.4496242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4496432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4496495Z res = mod(**inputs) 2025-12-04T09:44:06.4496712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4496783Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4497015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4497094Z outputs = self.model.decoder( 2025-12-04T09:44:06.4497303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4497379Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4497611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4497684Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4497906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4497981Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4498234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4498313Z return func(*args, **kwargs) 2025-12-04T09:44:06.4498550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4498652Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4498655Z 2025-12-04T09:44:06.4498753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4498945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4499014Z res = mod(**inputs) 2025-12-04T09:44:06.4499250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4499326Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4499576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4499647Z outputs = self.model.decoder( 2025-12-04T09:44:06.4499868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4499938Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4500172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4500251Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4500467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4500553Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4500804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4500873Z return func(*args, **kwargs) 2025-12-04T09:44:06.4501114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4501190Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4501194Z 2025-12-04T09:44:06.4501299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4501489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4501551Z res = mod(**inputs) 2025-12-04T09:44:06.4501768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4501836Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4502071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4502150Z outputs = self.model.decoder( 2025-12-04T09:44:06.4502366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4502442Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4502675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4502743Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4502964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4503039Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4503276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4503350Z return func(*args, **kwargs) 2025-12-04T09:44:06.4503589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4503691Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4503941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4504009Z return func(*args, **kwargs) 2025-12-04T09:44:06.4504252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4504361Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4504364Z 2025-12-04T09:44:06.4504470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4504660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4504721Z res = mod(**inputs) 2025-12-04T09:44:06.4504964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4505036Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4505292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4505372Z outputs = self.model.decoder( 2025-12-04T09:44:06.4505591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4505668Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4505901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4505970Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4506196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4506274Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4506525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4506601Z return func(*args, **kwargs) 2025-12-04T09:44:06.4506835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4506935Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4507167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4507234Z return func(*args, **kwargs) 2025-12-04T09:44:06.4507474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4507551Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4507554Z 2025-12-04T09:44:06.4507656Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4507855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4507913Z res = mod(**inputs) 2025-12-04T09:44:06.4508128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4508196Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4508421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4508494Z outputs = self.model.decoder( 2025-12-04T09:44:06.4508696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4508769Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4508992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4509061Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4509278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4509352Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4509592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4509669Z return func(*args, **kwargs) 2025-12-04T09:44:06.4509898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4509998Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4510225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4510287Z return func(*args, **kwargs) 2025-12-04T09:44:06.4510524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4510631Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4510635Z 2025-12-04T09:44:06.4510731Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4510809Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4510907Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4511105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4511164Z res = mod(**inputs) 2025-12-04T09:44:06.4511377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4511453Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4511682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4511758Z outputs = self.model.decoder( 2025-12-04T09:44:06.4511966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4512054Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4512291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4512359Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4512568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4512648Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4541791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4542038Z return func(*args, **kwargs) 2025-12-04T09:44:06.4542373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4542484Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4542782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4542867Z return func(*args, **kwargs) 2025-12-04T09:44:06.4543121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4543223Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4543524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4543659Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4543666Z 2025-12-04T09:44:06.4543785Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4543992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4544062Z res = mod(**inputs) 2025-12-04T09:44:06.4544295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4544375Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4544740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4544830Z outputs = self.model.decoder( 2025-12-04T09:44:06.4545050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4545130Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4545368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4545444Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4545674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4545796Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4546069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4546151Z return func(*args, **kwargs) 2025-12-04T09:44:06.4546389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4546500Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4546782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4546851Z return func(*args, **kwargs) 2025-12-04T09:44:06.4547094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4547178Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4547182Z 2025-12-04T09:44:06.4547297Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4547529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4547599Z res = mod(**inputs) 2025-12-04T09:44:06.4547827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4547901Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4548138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4548221Z outputs = self.model.decoder( 2025-12-04T09:44:06.4548435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4548513Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4548749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4548822Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4549049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4549126Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4549364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4549440Z return func(*args, **kwargs) 2025-12-04T09:44:06.4549677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4549764Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4549768Z 2025-12-04T09:44:06.4549870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4550067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4550137Z res = mod(**inputs) 2025-12-04T09:44:06.4550354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4550429Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4550686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4550761Z outputs = self.model.decoder( 2025-12-04T09:44:06.4550980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4551051Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4551287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4551367Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4551585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4551687Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4551925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4552008Z return func(*args, **kwargs) 2025-12-04T09:44:06.4552257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4552354Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4552358Z 2025-12-04T09:44:06.4552461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4552661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4552724Z res = mod(**inputs) 2025-12-04T09:44:06.4552945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4553016Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4553254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4553353Z outputs = self.model.decoder( 2025-12-04T09:44:06.4553574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4553651Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4553892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4553961Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4554189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4554263Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4554505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4554584Z return func(*args, **kwargs) 2025-12-04T09:44:06.4554825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4554914Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4554919Z 2025-12-04T09:44:06.4555020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4555219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4555290Z res = mod(**inputs) 2025-12-04T09:44:06.4555507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4555577Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4555828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4555901Z outputs = self.model.decoder( 2025-12-04T09:44:06.4556133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4556202Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4556458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4556536Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4556747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4556829Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4557058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4557122Z return func(*args, **kwargs) 2025-12-04T09:44:06.4557360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:44:06.4557488Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:44:06.4557514Z 2025-12-04T09:44:06.4557613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4557829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4557891Z res = mod(**inputs) 2025-12-04T09:44:06.4558106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4558176Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4558404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4558484Z outputs = self.model.decoder( 2025-12-04T09:44:06.4558690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4558762Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4559000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4559101Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4559320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4559393Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4559622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4559693Z return func(*args, **kwargs) 2025-12-04T09:44:06.4559924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4560028Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4560260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4560327Z return func(*args, **kwargs) 2025-12-04T09:44:06.4560566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4560678Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4560683Z 2025-12-04T09:44:06.4560783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4560981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4561043Z res = mod(**inputs) 2025-12-04T09:44:06.4561262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4561331Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4561562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4561642Z outputs = self.model.decoder( 2025-12-04T09:44:06.4561852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4561921Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4562179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4562250Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4562465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4562538Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4562766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4562837Z return func(*args, **kwargs) 2025-12-04T09:44:06.4563067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4563188Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4563421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4563501Z return func(*args, **kwargs) 2025-12-04T09:44:06.4563740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4563817Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4563821Z 2025-12-04T09:44:06.4563918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4564113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4564172Z res = mod(**inputs) 2025-12-04T09:44:06.4564382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4564452Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4564680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4564775Z outputs = self.model.decoder( 2025-12-04T09:44:06.4564993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4565062Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4565302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4565369Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4565588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4565661Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4565893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4565969Z return func(*args, **kwargs) 2025-12-04T09:44:06.4566204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4566311Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4566549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4566616Z return func(*args, **kwargs) 2025-12-04T09:44:06.4566861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4566944Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4566947Z 2025-12-04T09:44:06.4567029Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4567113Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4567211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4567415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4567479Z res = mod(**inputs) 2025-12-04T09:44:06.4567702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4567791Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4568034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4568113Z outputs = self.model.decoder( 2025-12-04T09:44:06.4568331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4568413Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4568659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4568729Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4568957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4569055Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4569314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4569392Z return func(*args, **kwargs) 2025-12-04T09:44:06.4569634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4569738Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4569980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4570046Z return func(*args, **kwargs) 2025-12-04T09:44:06.4570294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4570395Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4570711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4570855Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4570859Z 2025-12-04T09:44:06.4570959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4571158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4571315Z res = mod(**inputs) 2025-12-04T09:44:06.4571541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4571623Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4571868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4571952Z outputs = self.model.decoder( 2025-12-04T09:44:06.4572192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4572511Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4572799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4572877Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4573122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4573207Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4573445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4573520Z return func(*args, **kwargs) 2025-12-04T09:44:06.4573757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4573857Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4574105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4574240Z return func(*args, **kwargs) 2025-12-04T09:44:06.4574488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4574570Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4574573Z 2025-12-04T09:44:06.4574675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4574876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4574938Z res = mod(**inputs) 2025-12-04T09:44:06.4575151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4575233Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4575508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4576021Z outputs = self.model.decoder( 2025-12-04T09:44:06.4576244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4576317Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4576565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4576637Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4576862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4576948Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4577185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4577294Z return func(*args, **kwargs) 2025-12-04T09:44:06.4577540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4577623Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4577626Z 2025-12-04T09:44:06.4577739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4577935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4578009Z res = mod(**inputs) 2025-12-04T09:44:06.4578228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4578298Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4578544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4578615Z outputs = self.model.decoder( 2025-12-04T09:44:06.4578837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4578917Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4579157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4579234Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4579454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4579531Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4579779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4579846Z return func(*args, **kwargs) 2025-12-04T09:44:06.4580085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4580191Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4580196Z 2025-12-04T09:44:06.4580296Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4580518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4580583Z res = mod(**inputs) 2025-12-04T09:44:06.4580800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4580877Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4581117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4581195Z outputs = self.model.decoder( 2025-12-04T09:44:06.4581412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4581485Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4581762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4581836Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4582083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4582170Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4582413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4582488Z return func(*args, **kwargs) 2025-12-04T09:44:06.4582731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4582810Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4582814Z 2025-12-04T09:44:06.4582925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4583123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4583205Z res = mod(**inputs) 2025-12-04T09:44:06.4583439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4583514Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4583767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4583841Z outputs = self.model.decoder( 2025-12-04T09:44:06.4584066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4584140Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4584372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4584444Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4584658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4584734Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4584979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4585044Z return func(*args, **kwargs) 2025-12-04T09:44:06.4585285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4585387Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4585622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4585694Z return func(*args, **kwargs) 2025-12-04T09:44:06.4585927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4586036Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4586041Z 2025-12-04T09:44:06.4586144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4586352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4586422Z res = mod(**inputs) 2025-12-04T09:44:06.4586626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4586704Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4586933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4587001Z outputs = self.model.decoder( 2025-12-04T09:44:06.4587216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4587285Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4587533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4587617Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4587849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4587936Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4588180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4588243Z return func(*args, **kwargs) 2025-12-04T09:44:06.4588477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4588569Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4588798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4588871Z return func(*args, **kwargs) 2025-12-04T09:44:06.4589115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4589200Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4589205Z 2025-12-04T09:44:06.4589302Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4589488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4589555Z res = mod(**inputs) 2025-12-04T09:44:06.4589758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4589833Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4590064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4590135Z outputs = self.model.decoder( 2025-12-04T09:44:06.4590352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4590422Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4590657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4590734Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4590944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4591024Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4591254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4591320Z return func(*args, **kwargs) 2025-12-04T09:44:06.4591554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4591648Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4591881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4591955Z return func(*args, **kwargs) 2025-12-04T09:44:06.4592203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4592295Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4592299Z 2025-12-04T09:44:06.4592378Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4592455Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4592564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4592756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4592828Z res = mod(**inputs) 2025-12-04T09:44:06.4593041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4593133Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4593387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4593461Z outputs = self.model.decoder( 2025-12-04T09:44:06.4593670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4593747Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4593977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4594052Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4594263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4594338Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4594576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4594705Z return func(*args, **kwargs) 2025-12-04T09:44:06.4594936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4595035Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4595262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4595334Z return func(*args, **kwargs) 2025-12-04T09:44:06.4595563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4595655Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4595942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4596071Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4596075Z 2025-12-04T09:44:06.4596179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4596366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4596427Z res = mod(**inputs) 2025-12-04T09:44:06.4596638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4596707Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4596934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4597012Z outputs = self.model.decoder( 2025-12-04T09:44:06.4597220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4597298Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4597534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4597602Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4597841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4597919Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4598149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4598222Z return func(*args, **kwargs) 2025-12-04T09:44:06.4598454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4598556Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4598796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4598880Z return func(*args, **kwargs) 2025-12-04T09:44:06.4599129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4599210Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4599214Z 2025-12-04T09:44:06.4599315Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4599496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4599555Z res = mod(**inputs) 2025-12-04T09:44:06.4599765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4599831Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4600061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4600139Z outputs = self.model.decoder( 2025-12-04T09:44:06.4600371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4600445Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4600678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4600746Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4600967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4601042Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4601273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4601346Z return func(*args, **kwargs) 2025-12-04T09:44:06.4601575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4601662Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4601666Z 2025-12-04T09:44:06.4601764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4601955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4602024Z res = mod(**inputs) 2025-12-04T09:44:06.4602231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4602305Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4602537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4602605Z outputs = self.model.decoder( 2025-12-04T09:44:06.4602821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4602889Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4603123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4603201Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4603431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4603515Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4603750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4603815Z return func(*args, **kwargs) 2025-12-04T09:44:06.4604055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4604148Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4604151Z 2025-12-04T09:44:06.4604253Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4604444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4604540Z res = mod(**inputs) 2025-12-04T09:44:06.4604780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4604852Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4605084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4605161Z outputs = self.model.decoder( 2025-12-04T09:44:06.4605367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4605441Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4605674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4605741Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4605959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4606053Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4606289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4606361Z return func(*args, **kwargs) 2025-12-04T09:44:06.4606596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4606683Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4606686Z 2025-12-04T09:44:06.4606783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4606976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4607043Z res = mod(**inputs) 2025-12-04T09:44:06.4607256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4607333Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4607571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4607643Z outputs = self.model.decoder( 2025-12-04T09:44:06.4607872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4607940Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4608176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4608252Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4608467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4608550Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4608787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4608855Z return func(*args, **kwargs) 2025-12-04T09:44:06.4609117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T09:44:06.4609251Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T09:44:06.4609255Z 2025-12-04T09:44:06.4609356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4609560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4609621Z res = mod(**inputs) 2025-12-04T09:44:06.4609845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4609916Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4610155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4610256Z outputs = self.model.decoder( 2025-12-04T09:44:06.4610488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4610571Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4610811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4610880Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4611106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4611350Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4611603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4611681Z return func(*args, **kwargs) 2025-12-04T09:44:06.4611921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4612054Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4612299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4612368Z return func(*args, **kwargs) 2025-12-04T09:44:06.4612623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T09:44:06.4612737Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:44:06.4612741Z 2025-12-04T09:44:06.4612851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4613046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4613110Z res = mod(**inputs) 2025-12-04T09:44:06.4613336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4613412Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4613652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4613734Z outputs = self.model.decoder( 2025-12-04T09:44:06.4613950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4614028Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4614265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4614334Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4614571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4614646Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4614879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4614956Z return func(*args, **kwargs) 2025-12-04T09:44:06.4615211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4615317Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4615548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4615614Z return func(*args, **kwargs) 2025-12-04T09:44:06.4615856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T09:44:06.4615934Z key_states = self.k_proj(hidden_states) 2025-12-04T09:44:06.4615938Z 2025-12-04T09:44:06.4616042Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4616229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4616313Z res = mod(**inputs) 2025-12-04T09:44:06.4616554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4616627Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4616860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4616937Z outputs = self.model.decoder( 2025-12-04T09:44:06.4617149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4617224Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4617454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4617525Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4617745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4617841Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4618074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4618147Z return func(*args, **kwargs) 2025-12-04T09:44:06.4618388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4618491Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4618722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4618794Z return func(*args, **kwargs) 2025-12-04T09:44:06.4619026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T09:44:06.4619109Z value_states = self.v_proj(hidden_states) 2025-12-04T09:44:06.4619115Z 2025-12-04T09:44:06.4619203Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4619278Z cudagraph partition due to non gpu ops 2025-12-04T09:44:06.4619380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4619577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4619638Z res = mod(**inputs) 2025-12-04T09:44:06.4619854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4619923Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4620155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4620234Z outputs = self.model.decoder( 2025-12-04T09:44:06.4620442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4620514Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4620765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4620854Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4621079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4621152Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4621381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4621455Z return func(*args, **kwargs) 2025-12-04T09:44:06.4621683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4621792Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4622017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4622101Z return func(*args, **kwargs) 2025-12-04T09:44:06.4622352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T09:44:06.4622448Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:06.4622723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:06.4622856Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:06.4622859Z 2025-12-04T09:44:06.4622954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4623151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4623211Z res = mod(**inputs) 2025-12-04T09:44:06.4623418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4623515Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4623748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4623822Z outputs = self.model.decoder( 2025-12-04T09:44:06.4624026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4624093Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4624331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4624397Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4624604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4624681Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4624911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4624981Z return func(*args, **kwargs) 2025-12-04T09:44:06.4625212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T09:44:06.4625303Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:06.4625537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4625601Z return func(*args, **kwargs) 2025-12-04T09:44:06.4625827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T09:44:06.4625911Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:06.4625915Z 2025-12-04T09:44:06.4626010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4626201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4626262Z res = mod(**inputs) 2025-12-04T09:44:06.4626500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4626581Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4626811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4626887Z outputs = self.model.decoder( 2025-12-04T09:44:06.4627097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4627165Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4627400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4627467Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4627697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4627778Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4628023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4628097Z return func(*args, **kwargs) 2025-12-04T09:44:06.4628334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T09:44:06.4628409Z hidden_states = self.fc1(hidden_states) 2025-12-04T09:44:06.4628413Z 2025-12-04T09:44:06.4628516Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4628703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4628761Z res = mod(**inputs) 2025-12-04T09:44:06.4629021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4629114Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4629363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4629435Z outputs = self.model.decoder( 2025-12-04T09:44:06.4629642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4629719Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4629952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4630027Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4630239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4630314Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4630555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4630624Z return func(*args, **kwargs) 2025-12-04T09:44:06.4630861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T09:44:06.4630961Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T09:44:06.4630965Z 2025-12-04T09:44:06.4631061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4631258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4631319Z res = mod(**inputs) 2025-12-04T09:44:06.4631528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4631604Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4631836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T09:44:06.4631910Z outputs = self.model.decoder( 2025-12-04T09:44:06.4632130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4632214Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4632453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T09:44:06.4632523Z layer_outputs = decoder_layer( 2025-12-04T09:44:06.4632733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:06.4632814Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:06.4633048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:06.4633121Z return func(*args, **kwargs) 2025-12-04T09:44:06.4633350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T09:44:06.4633445Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:06.4633449Z 2025-12-04T09:44:06.4633570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4633762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4633823Z res = mod(**inputs) 2025-12-04T09:44:06.4634038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4634109Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4634348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 833, in forward 2025-12-04T09:44:06.4634439Z logits = self.lm_head(outputs[0]).contiguous() 2025-12-04T09:44:06.4634443Z 2025-12-04T09:44:06.4634541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:06.4634758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:06.4634819Z res = mod(**inputs) 2025-12-04T09:44:06.4635035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T09:44:06.4635103Z output = func(self, *args, **kwargs) 2025-12-04T09:44:06.4635342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 839, in forward 2025-12-04T09:44:06.4635419Z loss = self.loss_function( 2025-12-04T09:44:06.4635646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:44:06.4635810Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:44:06.4636060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:44:06.4636253Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:44:06.4636257Z 2025-12-04T09:44:17.9568399Z Compilation time (from dynamo_timed): 16.949096252 2025-12-04T09:44:17.9977585Z pass 2025-12-04T09:44:17.9978225Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:17.9979028Z TIMING: _recursive_pre_grad_passes:0.00803 _recursive_joint_graph_passes:0.62861 _recursive_post_grad_passes:0.08022 async_compile.wait:0.84024 code_gen:10.34707 inductor_compile:11.55092 backend_compile:14.53496 gc:0.00119 entire_frame_compile:16.9491 total_wall_time:16.9491 2025-12-04T09:44:17.9979961Z STATS: call_* op count: 379 | FakeTensorMode.__torch_dispatch__:7020 | FakeTensor.__torch_dispatch__:4035 | ProxyTorchDispatchMode.__torch_dispatch__:1957 2025-12-04T09:44:17.9980459Z Dynamo produced 1 graphs covering 379 ops with 0 graph breaks (0 unique) 2025-12-04T09:44:20.4208432Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:44:20.4209261Z import pynvml # type: ignore[import] 2025-12-04T09:44:23.8167749Z 2025-12-04T09:44:24.8553223Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:44:24.8553522Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:44:24.8565100Z cpu eval PLBartForCausalLM 2025-12-04T09:44:25.5192357Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:25.8153812Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:26.1086732Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:30.6203584Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6203927Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6204163Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6204732Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6204974Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6205190Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6205452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6205873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6206237Z res = mod(**inputs) 2025-12-04T09:44:30.6206684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6207143Z outputs = self.model.decoder( 2025-12-04T09:44:30.6207586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6208093Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6208517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6208939Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6209379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6209797Z return func(*args, **kwargs) 2025-12-04T09:44:30.6210234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6210715Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6211336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6211776Z return func(*args, **kwargs) 2025-12-04T09:44:30.6212210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:44:30.6212745Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:30.6212984Z 2025-12-04T09:44:30.6213103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6213499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6213846Z res = mod(**inputs) 2025-12-04T09:44:30.6214246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6214685Z outputs = self.model.decoder( 2025-12-04T09:44:30.6215116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6215605Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6215984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6216396Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6216893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6217313Z return func(*args, **kwargs) 2025-12-04T09:44:30.6217730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6218186Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6218614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6219009Z return func(*args, **kwargs) 2025-12-04T09:44:30.6219423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:44:30.6219867Z key_states = self.k_proj(current_states) 2025-12-04T09:44:30.6220084Z 2025-12-04T09:44:30.6220205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6220603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6220965Z res = mod(**inputs) 2025-12-04T09:44:30.6221366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6221794Z outputs = self.model.decoder( 2025-12-04T09:44:30.6222209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6222768Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6223153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6223551Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6223963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6224408Z return func(*args, **kwargs) 2025-12-04T09:44:30.6224832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6225309Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6225773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6226186Z return func(*args, **kwargs) 2025-12-04T09:44:30.6226629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:44:30.6227082Z value_states = self.v_proj(current_states) 2025-12-04T09:44:30.6227242Z 2025-12-04T09:44:30.6227334Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6227603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6228003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6228361Z res = mod(**inputs) 2025-12-04T09:44:30.6228765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6229203Z outputs = self.model.decoder( 2025-12-04T09:44:30.6229644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6230100Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6230490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6230897Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6231323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6231730Z return func(*args, **kwargs) 2025-12-04T09:44:30.6232149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6232632Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6233070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6233465Z return func(*args, **kwargs) 2025-12-04T09:44:30.6233893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:44:30.6234353Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:30.6234824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:30.6235337Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:30.6235568Z 2025-12-04T09:44:30.6235681Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6236090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6236429Z res = mod(**inputs) 2025-12-04T09:44:30.6236822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6237242Z outputs = self.model.decoder( 2025-12-04T09:44:30.6237651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6238060Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6238431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6238818Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6239209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6239626Z return func(*args, **kwargs) 2025-12-04T09:44:30.6240040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6240503Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6240935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6241338Z return func(*args, **kwargs) 2025-12-04T09:44:30.6241755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:44:30.6242211Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:30.6242361Z 2025-12-04T09:44:30.6242475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6242867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6243220Z res = mod(**inputs) 2025-12-04T09:44:30.6243616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6244049Z outputs = self.model.decoder( 2025-12-04T09:44:30.6244459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6244877Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6245252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6245649Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6246058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6246456Z return func(*args, **kwargs) 2025-12-04T09:44:30.6246870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6247357Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6247550Z 2025-12-04T09:44:30.6247690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6248073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6248424Z res = mod(**inputs) 2025-12-04T09:44:30.6248930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6249412Z outputs = self.model.decoder( 2025-12-04T09:44:30.6249828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6250271Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6250655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6251101Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6251612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6252022Z return func(*args, **kwargs) 2025-12-04T09:44:30.6252437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6252906Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6253328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:30.6253701Z return self.act(input) 2025-12-04T09:44:30.6253822Z 2025-12-04T09:44:30.6253934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6254318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6254694Z res = mod(**inputs) 2025-12-04T09:44:30.6255110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6255538Z outputs = self.model.decoder( 2025-12-04T09:44:30.6255960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6256393Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6256770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6257171Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6257587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6257992Z return func(*args, **kwargs) 2025-12-04T09:44:30.6258400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:44:30.6258843Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:30.6258998Z 2025-12-04T09:44:30.6259113Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6259506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6259850Z res = mod(**inputs) 2025-12-04T09:44:30.6260249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6260682Z outputs = self.model.decoder( 2025-12-04T09:44:30.6261096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6261525Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6261910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6262317Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6262734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6263232Z return func(*args, **kwargs) 2025-12-04T09:44:30.6263662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6264128Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6264555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6264948Z return func(*args, **kwargs) 2025-12-04T09:44:30.6265348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:44:30.6265836Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:30.6266090Z 2025-12-04T09:44:30.6266200Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6266596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6266945Z res = mod(**inputs) 2025-12-04T09:44:30.6267329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6267750Z outputs = self.model.decoder( 2025-12-04T09:44:30.6268172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6268595Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6268981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6269387Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6269810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6270229Z return func(*args, **kwargs) 2025-12-04T09:44:30.6270661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6271127Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6271570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6271974Z return func(*args, **kwargs) 2025-12-04T09:44:30.6272551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:44:30.6272992Z key_states = self.k_proj(current_states) 2025-12-04T09:44:30.6273142Z 2025-12-04T09:44:30.6273257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6273652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6274011Z res = mod(**inputs) 2025-12-04T09:44:30.6274421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6274856Z outputs = self.model.decoder( 2025-12-04T09:44:30.6275284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6275718Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6276096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6276496Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6276912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6277330Z return func(*args, **kwargs) 2025-12-04T09:44:30.6277743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6278206Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6278706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6279122Z return func(*args, **kwargs) 2025-12-04T09:44:30.6279528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:44:30.6279974Z value_states = self.v_proj(current_states) 2025-12-04T09:44:30.6280129Z 2025-12-04T09:44:30.6280227Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6280482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6280875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6281229Z res = mod(**inputs) 2025-12-04T09:44:30.6281664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6282116Z outputs = self.model.decoder( 2025-12-04T09:44:30.6282545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6282977Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6283350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6283742Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6284147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6284547Z return func(*args, **kwargs) 2025-12-04T09:44:30.6284950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6285438Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6285875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6286278Z return func(*args, **kwargs) 2025-12-04T09:44:30.6286681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:44:30.6287140Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:30.6287631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:30.6288154Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:30.6288364Z 2025-12-04T09:44:30.6288475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6288869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6289223Z res = mod(**inputs) 2025-12-04T09:44:30.6289619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6290050Z outputs = self.model.decoder( 2025-12-04T09:44:30.6290471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6290896Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6291346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6291763Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6292179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6292585Z return func(*args, **kwargs) 2025-12-04T09:44:30.6293009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6293473Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6293941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6294344Z return func(*args, **kwargs) 2025-12-04T09:44:30.6294760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:44:30.6295199Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:30.6295347Z 2025-12-04T09:44:30.6295458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6295849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6296214Z res = mod(**inputs) 2025-12-04T09:44:30.6296613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6297062Z outputs = self.model.decoder( 2025-12-04T09:44:30.6297514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6297937Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6298312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6298696Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6299098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6299496Z return func(*args, **kwargs) 2025-12-04T09:44:30.6299891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6300362Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6300578Z 2025-12-04T09:44:30.6300689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6301068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6301399Z res = mod(**inputs) 2025-12-04T09:44:30.6301786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6302201Z outputs = self.model.decoder( 2025-12-04T09:44:30.6302603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6303046Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6303417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6303802Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6304193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6304588Z return func(*args, **kwargs) 2025-12-04T09:44:30.6304993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6305453Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6305859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:30.6306225Z return self.act(input) 2025-12-04T09:44:30.6306342Z 2025-12-04T09:44:30.6306460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6306832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6307171Z res = mod(**inputs) 2025-12-04T09:44:30.6307560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6307982Z outputs = self.model.decoder( 2025-12-04T09:44:30.6308390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6308845Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6309219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6309599Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6310001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6310396Z return func(*args, **kwargs) 2025-12-04T09:44:30.6310798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:44:30.6311217Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:30.6311393Z 2025-12-04T09:44:30.6311505Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6311887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6312212Z res = mod(**inputs) 2025-12-04T09:44:30.6312573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6312958Z outputs = self.model.decoder( 2025-12-04T09:44:30.6313332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6313709Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6314050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6314405Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6314775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6315160Z return func(*args, **kwargs) 2025-12-04T09:44:30.6315533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6315943Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6316328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6316690Z return func(*args, **kwargs) 2025-12-04T09:44:30.6317060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:44:30.6317519Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:30.6317717Z 2025-12-04T09:44:30.6317820Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6318167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6318481Z res = mod(**inputs) 2025-12-04T09:44:30.6318835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6319216Z outputs = self.model.decoder( 2025-12-04T09:44:30.6319593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6319978Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6320310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6320664Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6321030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6321394Z return func(*args, **kwargs) 2025-12-04T09:44:30.6321755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6322161Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6322575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6322944Z return func(*args, **kwargs) 2025-12-04T09:44:30.6323312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:44:30.6323709Z key_states = self.k_proj(current_states) 2025-12-04T09:44:30.6323842Z 2025-12-04T09:44:30.6323954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6324299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6324619Z res = mod(**inputs) 2025-12-04T09:44:30.6324979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6325381Z outputs = self.model.decoder( 2025-12-04T09:44:30.6325772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6326153Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6326490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6326834Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6327201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6327562Z return func(*args, **kwargs) 2025-12-04T09:44:30.6327927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6328321Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6328768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6329143Z return func(*args, **kwargs) 2025-12-04T09:44:30.6329527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:44:30.6329923Z value_states = self.v_proj(current_states) 2025-12-04T09:44:30.6330070Z 2025-12-04T09:44:30.6330150Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6330391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6330738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6331058Z res = mod(**inputs) 2025-12-04T09:44:30.6331547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6331994Z outputs = self.model.decoder( 2025-12-04T09:44:30.6332415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6332851Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6333212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6333573Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6333961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6334342Z return func(*args, **kwargs) 2025-12-04T09:44:30.6334732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6335148Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6335557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6335938Z return func(*args, **kwargs) 2025-12-04T09:44:30.6336325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:44:30.6336758Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:30.6337211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:30.6337697Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:30.6337882Z 2025-12-04T09:44:30.6337983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6338341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6338660Z res = mod(**inputs) 2025-12-04T09:44:30.6339026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6339432Z outputs = self.model.decoder( 2025-12-04T09:44:30.6339838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6340236Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6340742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6341115Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6341498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6341876Z return func(*args, **kwargs) 2025-12-04T09:44:30.6342252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6342675Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6343108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6343482Z return func(*args, **kwargs) 2025-12-04T09:44:30.6343859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:44:30.6344265Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:30.6344402Z 2025-12-04T09:44:30.6344514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6344865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6345188Z res = mod(**inputs) 2025-12-04T09:44:30.6345551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6345945Z outputs = self.model.decoder( 2025-12-04T09:44:30.6346330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6346725Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6347079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6347440Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6347812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6348188Z return func(*args, **kwargs) 2025-12-04T09:44:30.6348567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6349003Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6349179Z 2025-12-04T09:44:30.6349294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6349642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6349958Z res = mod(**inputs) 2025-12-04T09:44:30.6350313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6350723Z outputs = self.model.decoder( 2025-12-04T09:44:30.6351112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6351501Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6351855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6352221Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6352605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6352975Z return func(*args, **kwargs) 2025-12-04T09:44:30.6353381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6353839Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6354236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:30.6354580Z return self.act(input) 2025-12-04T09:44:30.6354700Z 2025-12-04T09:44:30.6354806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6355164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6355483Z res = mod(**inputs) 2025-12-04T09:44:30.6355854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6356302Z outputs = self.model.decoder( 2025-12-04T09:44:30.6356683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6357090Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6357429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6357789Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6358149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6358513Z return func(*args, **kwargs) 2025-12-04T09:44:30.6358878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:44:30.6359266Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:30.6359400Z 2025-12-04T09:44:30.6359501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6359846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6360163Z res = mod(**inputs) 2025-12-04T09:44:30.6360514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6360899Z outputs = self.model.decoder( 2025-12-04T09:44:30.6361275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6361661Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6361992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6362342Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6362710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6363071Z return func(*args, **kwargs) 2025-12-04T09:44:30.6363431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6363841Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6364247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6364603Z return func(*args, **kwargs) 2025-12-04T09:44:30.6364971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:44:30.6365428Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:30.6365624Z 2025-12-04T09:44:30.6365730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6366068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6366382Z res = mod(**inputs) 2025-12-04T09:44:30.6366746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6367139Z outputs = self.model.decoder( 2025-12-04T09:44:30.6367514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6367887Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6368214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6368547Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6368907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6369260Z return func(*args, **kwargs) 2025-12-04T09:44:30.6369617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6370004Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6370410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6370771Z return func(*args, **kwargs) 2025-12-04T09:44:30.6371129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:44:30.6371647Z key_states = self.k_proj(current_states) 2025-12-04T09:44:30.6371808Z 2025-12-04T09:44:30.6371925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6372443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6372805Z res = mod(**inputs) 2025-12-04T09:44:30.6373197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6373595Z outputs = self.model.decoder( 2025-12-04T09:44:30.6374003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6374387Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6374731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6375089Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6375453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6375817Z return func(*args, **kwargs) 2025-12-04T09:44:30.6376192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6376598Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6376980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6377342Z return func(*args, **kwargs) 2025-12-04T09:44:30.6377719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:44:30.6378124Z value_states = self.v_proj(current_states) 2025-12-04T09:44:30.6378327Z 2025-12-04T09:44:30.6378412Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6378656Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6379019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6379336Z res = mod(**inputs) 2025-12-04T09:44:30.6379705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6380102Z outputs = self.model.decoder( 2025-12-04T09:44:30.6380492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6380911Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6381256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6381640Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6382014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6382389Z return func(*args, **kwargs) 2025-12-04T09:44:30.6382766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6383176Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6383566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6383936Z return func(*args, **kwargs) 2025-12-04T09:44:30.6384313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:44:30.6384751Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:30.6385196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:30.6385678Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:30.6385859Z 2025-12-04T09:44:30.6385970Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6386319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6386641Z res = mod(**inputs) 2025-12-04T09:44:30.6387005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6387397Z outputs = self.model.decoder( 2025-12-04T09:44:30.6387777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6388172Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6388525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6388884Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6389258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6389633Z return func(*args, **kwargs) 2025-12-04T09:44:30.6390007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6390418Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6390813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6391182Z return func(*args, **kwargs) 2025-12-04T09:44:30.6391564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:44:30.6391965Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:30.6392109Z 2025-12-04T09:44:30.6392229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6392591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6392908Z res = mod(**inputs) 2025-12-04T09:44:30.6393277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6393672Z outputs = self.model.decoder( 2025-12-04T09:44:30.6394059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6394444Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6394822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6395192Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6395589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6395962Z return func(*args, **kwargs) 2025-12-04T09:44:30.6396340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6396777Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6396946Z 2025-12-04T09:44:30.6397050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6397408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6397727Z res = mod(**inputs) 2025-12-04T09:44:30.6398094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6398513Z outputs = self.model.decoder( 2025-12-04T09:44:30.6398889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6399270Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6399601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6399954Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6400331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6400706Z return func(*args, **kwargs) 2025-12-04T09:44:30.6401079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6401515Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6401906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:30.6402240Z return self.act(input) 2025-12-04T09:44:30.6402347Z 2025-12-04T09:44:30.6402450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6402799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6403110Z res = mod(**inputs) 2025-12-04T09:44:30.6403459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6403846Z outputs = self.model.decoder( 2025-12-04T09:44:30.6404217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6404608Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6404949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6405315Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6405717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6406090Z return func(*args, **kwargs) 2025-12-04T09:44:30.6406476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:44:30.6406880Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:30.6407018Z 2025-12-04T09:44:30.6407130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6407480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6407803Z res = mod(**inputs) 2025-12-04T09:44:30.6408173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6408579Z outputs = self.model.decoder( 2025-12-04T09:44:30.6408990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6409392Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6409747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6410104Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6410512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6410926Z return func(*args, **kwargs) 2025-12-04T09:44:30.6411406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6411850Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6412306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6412729Z return func(*args, **kwargs) 2025-12-04T09:44:30.6413130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:44:30.6413688Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:30.6413913Z 2025-12-04T09:44:30.6414024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6414401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6414733Z res = mod(**inputs) 2025-12-04T09:44:30.6415123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6415541Z outputs = self.model.decoder( 2025-12-04T09:44:30.6415946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6416356Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6416728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6417110Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6417504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6417898Z return func(*args, **kwargs) 2025-12-04T09:44:30.6418296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6418733Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6419145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6419540Z return func(*args, **kwargs) 2025-12-04T09:44:30.6419942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:44:30.6420357Z key_states = self.k_proj(current_states) 2025-12-04T09:44:30.6420534Z 2025-12-04T09:44:30.6420647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6421024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6421362Z res = mod(**inputs) 2025-12-04T09:44:30.6421741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6422159Z outputs = self.model.decoder( 2025-12-04T09:44:30.6422566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6422979Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6423370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6423752Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6424169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6424562Z return func(*args, **kwargs) 2025-12-04T09:44:30.6424979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6425430Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6425861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6426247Z return func(*args, **kwargs) 2025-12-04T09:44:30.6426625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:44:30.6427058Z value_states = self.v_proj(current_states) 2025-12-04T09:44:30.6427201Z 2025-12-04T09:44:30.6427291Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6427526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6427881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6428202Z res = mod(**inputs) 2025-12-04T09:44:30.6428560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6428957Z outputs = self.model.decoder( 2025-12-04T09:44:30.6429346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6429738Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6430077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6430448Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6430817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6431170Z return func(*args, **kwargs) 2025-12-04T09:44:30.6431536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6431938Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6432322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6432671Z return func(*args, **kwargs) 2025-12-04T09:44:30.6433036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:44:30.6433438Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:30.6433867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:30.6434338Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:30.6434522Z 2025-12-04T09:44:30.6434654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6435006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6435312Z res = mod(**inputs) 2025-12-04T09:44:30.6435667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6436048Z outputs = self.model.decoder( 2025-12-04T09:44:30.6436425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6436803Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6437147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6437524Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6437905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6438275Z return func(*args, **kwargs) 2025-12-04T09:44:30.6438650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6439056Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6439437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6439804Z return func(*args, **kwargs) 2025-12-04T09:44:30.6440176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:44:30.6440574Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:30.6440726Z 2025-12-04T09:44:30.6440825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6441179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6441500Z res = mod(**inputs) 2025-12-04T09:44:30.6441851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6442239Z outputs = self.model.decoder( 2025-12-04T09:44:30.6442619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6443007Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6443350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6443709Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6444076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6444425Z return func(*args, **kwargs) 2025-12-04T09:44:30.6444788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6445209Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6445373Z 2025-12-04T09:44:30.6445483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6445825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6446135Z res = mod(**inputs) 2025-12-04T09:44:30.6446493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6446879Z outputs = self.model.decoder( 2025-12-04T09:44:30.6447252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6447645Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6448008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6448356Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6448727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6449089Z return func(*args, **kwargs) 2025-12-04T09:44:30.6449459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6449877Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6450254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:30.6450606Z return self.act(input) 2025-12-04T09:44:30.6450714Z 2025-12-04T09:44:30.6450815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6451238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6451574Z res = mod(**inputs) 2025-12-04T09:44:30.6451954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6452370Z outputs = self.model.decoder( 2025-12-04T09:44:30.6452780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6453196Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6453568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6453944Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6454350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6454770Z return func(*args, **kwargs) 2025-12-04T09:44:30.6455167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:44:30.6455606Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:30.6455750Z 2025-12-04T09:44:30.6455854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6456211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6456526Z res = mod(**inputs) 2025-12-04T09:44:30.6456890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6457286Z outputs = self.model.decoder( 2025-12-04T09:44:30.6457667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6458062Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6458413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6458775Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6459152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6459524Z return func(*args, **kwargs) 2025-12-04T09:44:30.6459906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6460328Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6460720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6461149Z return func(*args, **kwargs) 2025-12-04T09:44:30.6461529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T09:44:30.6461990Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:30.6462219Z 2025-12-04T09:44:30.6462326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6462686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6463006Z res = mod(**inputs) 2025-12-04T09:44:30.6463362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6463764Z outputs = self.model.decoder( 2025-12-04T09:44:30.6464157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6464552Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6464932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6465301Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6465706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6466075Z return func(*args, **kwargs) 2025-12-04T09:44:30.6466457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6466876Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6467274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6467636Z return func(*args, **kwargs) 2025-12-04T09:44:30.6468011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T09:44:30.6468434Z key_states = self.k_proj(current_states) 2025-12-04T09:44:30.6468568Z 2025-12-04T09:44:30.6468673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6469033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6469352Z res = mod(**inputs) 2025-12-04T09:44:30.6469715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6470100Z outputs = self.model.decoder( 2025-12-04T09:44:30.6470488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6470879Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6471219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6471580Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6471960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6472523Z return func(*args, **kwargs) 2025-12-04T09:44:30.6472894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6473323Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6473736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6474101Z return func(*args, **kwargs) 2025-12-04T09:44:30.6474464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T09:44:30.6474917Z value_states = self.v_proj(current_states) 2025-12-04T09:44:30.6475062Z 2025-12-04T09:44:30.6475141Z cudagraph partition due to non gpu ops 2025-12-04T09:44:30.6475382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6475721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6476040Z res = mod(**inputs) 2025-12-04T09:44:30.6476448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6476835Z outputs = self.model.decoder( 2025-12-04T09:44:30.6477204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6477597Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6477949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6478318Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6478682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6479087Z return func(*args, **kwargs) 2025-12-04T09:44:30.6479482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6479953Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6480351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6480724Z return func(*args, **kwargs) 2025-12-04T09:44:30.6481103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T09:44:30.6481513Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:30.6481963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:30.6482448Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:30.6482664Z 2025-12-04T09:44:30.6482774Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6483127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6483455Z res = mod(**inputs) 2025-12-04T09:44:30.6483824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6484220Z outputs = self.model.decoder( 2025-12-04T09:44:30.6484606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6484998Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6485350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6485706Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6486087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6486457Z return func(*args, **kwargs) 2025-12-04T09:44:30.6486831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T09:44:30.6487254Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:30.6487647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6488017Z return func(*args, **kwargs) 2025-12-04T09:44:30.6488386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T09:44:30.6488787Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:30.6488920Z 2025-12-04T09:44:30.6489034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6489393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6489709Z res = mod(**inputs) 2025-12-04T09:44:30.6490093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6490490Z outputs = self.model.decoder( 2025-12-04T09:44:30.6490876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6491366Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6491755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6492162Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6492555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6492952Z return func(*args, **kwargs) 2025-12-04T09:44:30.6493366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6493811Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6493996Z 2025-12-04T09:44:30.6494103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6494461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6494783Z res = mod(**inputs) 2025-12-04T09:44:30.6495140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6495532Z outputs = self.model.decoder( 2025-12-04T09:44:30.6495918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6496312Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6496662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6497054Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6497441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6497810Z return func(*args, **kwargs) 2025-12-04T09:44:30.6498196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T09:44:30.6498636Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:30.6499029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:30.6499371Z return self.act(input) 2025-12-04T09:44:30.6499487Z 2025-12-04T09:44:30.6499591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6499954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6500275Z res = mod(**inputs) 2025-12-04T09:44:30.6500650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T09:44:30.6501060Z outputs = self.model.decoder( 2025-12-04T09:44:30.6501451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T09:44:30.6501841Z layer_outputs = decoder_layer( 2025-12-04T09:44:30.6502196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:30.6502563Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:30.6502941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:30.6503321Z return func(*args, **kwargs) 2025-12-04T09:44:30.6503705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T09:44:30.6504119Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:30.6504258Z 2025-12-04T09:44:30.6504389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6504743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6505060Z res = mod(**inputs) 2025-12-04T09:44:30.6505422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1696, in forward 2025-12-04T09:44:30.6505810Z logits = self.lm_head(outputs[0]) 2025-12-04T09:44:30.6505946Z 2025-12-04T09:44:30.6506046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:30.6506397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:30.6506706Z res = mod(**inputs) 2025-12-04T09:44:30.6507094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1702, in forward 2025-12-04T09:44:30.6507580Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:44:30.6507780Z 2025-12-04T09:44:39.8668826Z Compilation time (from dynamo_timed): 12.91200244 2025-12-04T09:44:39.8956757Z pass 2025-12-04T09:44:39.8957272Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:39.8958093Z TIMING: _recursive_pre_grad_passes:0.00532 _recursive_joint_graph_passes:0.25422 _recursive_post_grad_passes:0.04578 async_compile.wait:0.77931 code_gen:8.99662 inductor_compile:9.70237 backend_compile:11.44713 gc:0.00104 entire_frame_compile:12.912 total_wall_time:12.912 2025-12-04T09:44:39.8959009Z STATS: call_* op count: 180 | FakeTensorMode.__torch_dispatch__:4080 | FakeTensor.__torch_dispatch__:2342 | ProxyTorchDispatchMode.__torch_dispatch__:1092 2025-12-04T09:44:39.8959819Z Dynamo produced 1 graphs covering 180 ops with 0 graph breaks (0 unique) 2025-12-04T09:44:42.1421397Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:44:42.1422353Z import pynvml # type: ignore[import] 2025-12-04T09:44:45.4696930Z 2025-12-04T09:44:48.5147336Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:44:48.5147618Z loading model: 0it [00:03, ?it/s] 2025-12-04T09:44:48.5168295Z cpu eval PegasusForCausalLM 2025-12-04T09:44:48.8718214Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:49.0165520Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:49.1468309Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:44:56.1040657Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1047527Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1049537Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1049772Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1050343Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1050626Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1050863Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1051091Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1051376Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1051614Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1051842Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1052069Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1052342Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1052793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1053135Z res = mod(**inputs) 2025-12-04T09:44:56.1053895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1054368Z outputs = self.model.decoder( 2025-12-04T09:44:56.1054800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1055227Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1055611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1056019Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1056427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1056825Z return func(*args, **kwargs) 2025-12-04T09:44:56.1057337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1057892Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1058346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1058750Z return func(*args, **kwargs) 2025-12-04T09:44:56.1059138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1059647Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1059873Z 2025-12-04T09:44:56.1059982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1060358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1060679Z res = mod(**inputs) 2025-12-04T09:44:56.1061122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1061548Z outputs = self.model.decoder( 2025-12-04T09:44:56.1061950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1062362Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1062716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1063088Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1063480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1063873Z return func(*args, **kwargs) 2025-12-04T09:44:56.1064271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1064718Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1065106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1065467Z return func(*args, **kwargs) 2025-12-04T09:44:56.1065849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1066246Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1066388Z 2025-12-04T09:44:56.1066493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1066873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1067196Z res = mod(**inputs) 2025-12-04T09:44:56.1067558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1067956Z outputs = self.model.decoder( 2025-12-04T09:44:56.1068345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1068743Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1069114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1069490Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1069870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1070248Z return func(*args, **kwargs) 2025-12-04T09:44:56.1070639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1071069Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1071469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1071848Z return func(*args, **kwargs) 2025-12-04T09:44:56.1072413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1072854Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1072998Z 2025-12-04T09:44:56.1073093Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1073326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1073690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1074014Z res = mod(**inputs) 2025-12-04T09:44:56.1074381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1074789Z outputs = self.model.decoder( 2025-12-04T09:44:56.1075189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1075630Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1075969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1076330Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1076697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1077054Z return func(*args, **kwargs) 2025-12-04T09:44:56.1077429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1077843Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1078230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1078587Z return func(*args, **kwargs) 2025-12-04T09:44:56.1078963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1079377Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1079817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1080287Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1080475Z 2025-12-04T09:44:56.1080576Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1080926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1081231Z res = mod(**inputs) 2025-12-04T09:44:56.1081594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1081983Z outputs = self.model.decoder( 2025-12-04T09:44:56.1082366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1082745Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1083111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1083470Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1083847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1084221Z return func(*args, **kwargs) 2025-12-04T09:44:56.1084591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1085009Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1085400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1085798Z return func(*args, **kwargs) 2025-12-04T09:44:56.1086206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1086610Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1086749Z 2025-12-04T09:44:56.1086853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1087213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1087540Z res = mod(**inputs) 2025-12-04T09:44:56.1087911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1088317Z outputs = self.model.decoder( 2025-12-04T09:44:56.1088721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1089134Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1089477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1089844Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1090222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1090593Z return func(*args, **kwargs) 2025-12-04T09:44:56.1090972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1091597Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1091787Z 2025-12-04T09:44:56.1091909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1092284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1092630Z res = mod(**inputs) 2025-12-04T09:44:56.1093032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1093464Z outputs = self.model.decoder( 2025-12-04T09:44:56.1093878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1094303Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1094683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1095059Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1095453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1095828Z return func(*args, **kwargs) 2025-12-04T09:44:56.1096218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1096661Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1097080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1097432Z return self.act(input) 2025-12-04T09:44:56.1097543Z 2025-12-04T09:44:56.1097654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1098005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1098353Z res = mod(**inputs) 2025-12-04T09:44:56.1098724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1099119Z outputs = self.model.decoder( 2025-12-04T09:44:56.1099507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1099926Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1100311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1100684Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1101062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1101430Z return func(*args, **kwargs) 2025-12-04T09:44:56.1101816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1102219Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1102359Z 2025-12-04T09:44:56.1102471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1102822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1103144Z res = mod(**inputs) 2025-12-04T09:44:56.1103548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1103958Z outputs = self.model.decoder( 2025-12-04T09:44:56.1104347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1104741Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1105080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1105428Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1105798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1106165Z return func(*args, **kwargs) 2025-12-04T09:44:56.1106537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1106955Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1107350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1107720Z return func(*args, **kwargs) 2025-12-04T09:44:56.1108093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1108559Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1108767Z 2025-12-04T09:44:56.1108867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1109216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1109521Z res = mod(**inputs) 2025-12-04T09:44:56.1109890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1110290Z outputs = self.model.decoder( 2025-12-04T09:44:56.1110670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1111085Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1111429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1111782Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1112144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1112510Z return func(*args, **kwargs) 2025-12-04T09:44:56.1112881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1113291Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1113695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1114300Z return func(*args, **kwargs) 2025-12-04T09:44:56.1114703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1115088Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1115232Z 2025-12-04T09:44:56.1115335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1115687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1116002Z res = mod(**inputs) 2025-12-04T09:44:56.1116359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1116750Z outputs = self.model.decoder( 2025-12-04T09:44:56.1117132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1117544Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1117879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1118235Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1118601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1118955Z return func(*args, **kwargs) 2025-12-04T09:44:56.1119328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1119750Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1120150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1120502Z return func(*args, **kwargs) 2025-12-04T09:44:56.1120879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1121294Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1121434Z 2025-12-04T09:44:56.1121523Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1121757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1122111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1122430Z res = mod(**inputs) 2025-12-04T09:44:56.1122789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1123188Z outputs = self.model.decoder( 2025-12-04T09:44:56.1123563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1123950Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1124371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1124727Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1125115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1125474Z return func(*args, **kwargs) 2025-12-04T09:44:56.1125848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1126262Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1126650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1127006Z return func(*args, **kwargs) 2025-12-04T09:44:56.1127373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1127807Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1128266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1128756Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1128956Z 2025-12-04T09:44:56.1129066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1129459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1129773Z res = mod(**inputs) 2025-12-04T09:44:56.1130145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1130543Z outputs = self.model.decoder( 2025-12-04T09:44:56.1130930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1131540Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1131937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1132357Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1132777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1133191Z return func(*args, **kwargs) 2025-12-04T09:44:56.1133606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1134061Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1134493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1134907Z return func(*args, **kwargs) 2025-12-04T09:44:56.1135323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1135761Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1135910Z 2025-12-04T09:44:56.1136023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1136412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1136768Z res = mod(**inputs) 2025-12-04T09:44:56.1137151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1137577Z outputs = self.model.decoder( 2025-12-04T09:44:56.1137994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1138418Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1138788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1139179Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1139609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1139996Z return func(*args, **kwargs) 2025-12-04T09:44:56.1140399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1140864Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1141047Z 2025-12-04T09:44:56.1141163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1141527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1141866Z res = mod(**inputs) 2025-12-04T09:44:56.1142255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1142697Z outputs = self.model.decoder( 2025-12-04T09:44:56.1143097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1143496Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1143860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1144247Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1144625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1144998Z return func(*args, **kwargs) 2025-12-04T09:44:56.1145379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1145839Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1146274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1146645Z return self.act(input) 2025-12-04T09:44:56.1146763Z 2025-12-04T09:44:56.1146880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1147253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1147594Z res = mod(**inputs) 2025-12-04T09:44:56.1147988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1148401Z outputs = self.model.decoder( 2025-12-04T09:44:56.1148815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1149234Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1149603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1149985Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1150397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1150794Z return func(*args, **kwargs) 2025-12-04T09:44:56.1151194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1151622Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1151773Z 2025-12-04T09:44:56.1151880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1152253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1152584Z res = mod(**inputs) 2025-12-04T09:44:56.1152976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1153403Z outputs = self.model.decoder( 2025-12-04T09:44:56.1153815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1154264Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1154639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1155024Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1155416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1155815Z return func(*args, **kwargs) 2025-12-04T09:44:56.1156221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1156669Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1157125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1157522Z return func(*args, **kwargs) 2025-12-04T09:44:56.1157946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1158442Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1158663Z 2025-12-04T09:44:56.1158773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1159154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1159470Z res = mod(**inputs) 2025-12-04T09:44:56.1159835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1160229Z outputs = self.model.decoder( 2025-12-04T09:44:56.1160620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1161041Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1161387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1161753Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1162128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1162493Z return func(*args, **kwargs) 2025-12-04T09:44:56.1162873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1163298Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1163694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1164063Z return func(*args, **kwargs) 2025-12-04T09:44:56.1164444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1164849Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1164982Z 2025-12-04T09:44:56.1165094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1165444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1165765Z res = mod(**inputs) 2025-12-04T09:44:56.1166136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1166529Z outputs = self.model.decoder( 2025-12-04T09:44:56.1166926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1167327Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1167676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1168034Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1168446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1168854Z return func(*args, **kwargs) 2025-12-04T09:44:56.1169254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1169717Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1170133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1170538Z return func(*args, **kwargs) 2025-12-04T09:44:56.1170933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1171540Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1171697Z 2025-12-04T09:44:56.1171799Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1172079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1172627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1172999Z res = mod(**inputs) 2025-12-04T09:44:56.1173403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1173812Z outputs = self.model.decoder( 2025-12-04T09:44:56.1174208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1174606Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1174957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1175380Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1175762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1176170Z return func(*args, **kwargs) 2025-12-04T09:44:56.1176579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1177030Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1177461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1177862Z return func(*args, **kwargs) 2025-12-04T09:44:56.1178258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1178710Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1179193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1179712Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1179909Z 2025-12-04T09:44:56.1180022Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1180399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1180738Z res = mod(**inputs) 2025-12-04T09:44:56.1181146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1181579Z outputs = self.model.decoder( 2025-12-04T09:44:56.1181995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1182423Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1182789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1183173Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1183607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1183996Z return func(*args, **kwargs) 2025-12-04T09:44:56.1184399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1184847Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1185270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1185653Z return func(*args, **kwargs) 2025-12-04T09:44:56.1186056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1186518Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1186663Z 2025-12-04T09:44:56.1186780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1187175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1187519Z res = mod(**inputs) 2025-12-04T09:44:56.1187911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1188328Z outputs = self.model.decoder( 2025-12-04T09:44:56.1188753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1189136Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1189472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1189818Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1190209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1190574Z return func(*args, **kwargs) 2025-12-04T09:44:56.1190937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1191363Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1191536Z 2025-12-04T09:44:56.1191637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1191982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1192289Z res = mod(**inputs) 2025-12-04T09:44:56.1192663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1193058Z outputs = self.model.decoder( 2025-12-04T09:44:56.1193451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1193841Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1194193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1194547Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1194907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1195269Z return func(*args, **kwargs) 2025-12-04T09:44:56.1195637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1196057Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1196424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1196770Z return self.act(input) 2025-12-04T09:44:56.1196876Z 2025-12-04T09:44:56.1196983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1197339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1197655Z res = mod(**inputs) 2025-12-04T09:44:56.1198021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1198410Z outputs = self.model.decoder( 2025-12-04T09:44:56.1198789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1199179Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1199520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1199877Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1200260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1200649Z return func(*args, **kwargs) 2025-12-04T09:44:56.1201026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1201422Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1201571Z 2025-12-04T09:44:56.1201676Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1202027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1202347Z res = mod(**inputs) 2025-12-04T09:44:56.1202709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1203112Z outputs = self.model.decoder( 2025-12-04T09:44:56.1203500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1203904Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1204247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1204604Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1204975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1205328Z return func(*args, **kwargs) 2025-12-04T09:44:56.1205698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:44:56.1206086Z hidden_states = residual + hidden_states 2025-12-04T09:44:56.1206215Z 2025-12-04T09:44:56.1206320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1206655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1206968Z res = mod(**inputs) 2025-12-04T09:44:56.1207326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1207707Z outputs = self.model.decoder( 2025-12-04T09:44:56.1208086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1208472Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1208811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1209152Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1209522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1209896Z return func(*args, **kwargs) 2025-12-04T09:44:56.1210270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1210688Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1211128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1211607Z return func(*args, **kwargs) 2025-12-04T09:44:56.1212022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1212551Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1212779Z 2025-12-04T09:44:56.1212890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1213291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1213659Z res = mod(**inputs) 2025-12-04T09:44:56.1214072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1214482Z outputs = self.model.decoder( 2025-12-04T09:44:56.1214891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1215291Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1215639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1216000Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1216372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1216741Z return func(*args, **kwargs) 2025-12-04T09:44:56.1217122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1217562Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1217955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1218332Z return func(*args, **kwargs) 2025-12-04T09:44:56.1218714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1219112Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1219253Z 2025-12-04T09:44:56.1219358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1219710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1220034Z res = mod(**inputs) 2025-12-04T09:44:56.1220398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1220798Z outputs = self.model.decoder( 2025-12-04T09:44:56.1221191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1221580Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1221927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1222292Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1222663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1223027Z return func(*args, **kwargs) 2025-12-04T09:44:56.1223409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1223825Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1224216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1224586Z return func(*args, **kwargs) 2025-12-04T09:44:56.1224968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1225398Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1225540Z 2025-12-04T09:44:56.1225621Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1225860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1226219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1226542Z res = mod(**inputs) 2025-12-04T09:44:56.1226907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1227318Z outputs = self.model.decoder( 2025-12-04T09:44:56.1227703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1228122Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1228488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1228855Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1229279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1229641Z return func(*args, **kwargs) 2025-12-04T09:44:56.1230022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1230428Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1230810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1231165Z return func(*args, **kwargs) 2025-12-04T09:44:56.1231565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1231979Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1232410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1232882Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1233065Z 2025-12-04T09:44:56.1233167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1233516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1233827Z res = mod(**inputs) 2025-12-04T09:44:56.1234189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1234585Z outputs = self.model.decoder( 2025-12-04T09:44:56.1234966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1235357Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1235707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1236070Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1236436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1236803Z return func(*args, **kwargs) 2025-12-04T09:44:56.1237178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1237591Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1237976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1238341Z return func(*args, **kwargs) 2025-12-04T09:44:56.1238716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1239124Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1239268Z 2025-12-04T09:44:56.1239369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1239718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1240032Z res = mod(**inputs) 2025-12-04T09:44:56.1240390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1240802Z outputs = self.model.decoder( 2025-12-04T09:44:56.1241180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1241586Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1241919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1242289Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1242665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1243024Z return func(*args, **kwargs) 2025-12-04T09:44:56.1243408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1243855Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1244027Z 2025-12-04T09:44:56.1244137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1244486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1244815Z res = mod(**inputs) 2025-12-04T09:44:56.1245236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1245666Z outputs = self.model.decoder( 2025-12-04T09:44:56.1246085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1246498Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1246846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1247204Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1247614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1248016Z return func(*args, **kwargs) 2025-12-04T09:44:56.1248418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1248891Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1249306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1249676Z return self.act(input) 2025-12-04T09:44:56.1249790Z 2025-12-04T09:44:56.1249897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1250286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1250631Z res = mod(**inputs) 2025-12-04T09:44:56.1251022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1251533Z outputs = self.model.decoder( 2025-12-04T09:44:56.1251971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1252447Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1252822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1253214Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1253629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1254010Z return func(*args, **kwargs) 2025-12-04T09:44:56.1254394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1254801Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1254938Z 2025-12-04T09:44:56.1255048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1255411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1255724Z res = mod(**inputs) 2025-12-04T09:44:56.1256123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1256525Z outputs = self.model.decoder( 2025-12-04T09:44:56.1256933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1257335Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1257684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1258046Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1258424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1258802Z return func(*args, **kwargs) 2025-12-04T09:44:56.1259183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1259631Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1260032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1260405Z return func(*args, **kwargs) 2025-12-04T09:44:56.1260792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1261255Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1261467Z 2025-12-04T09:44:56.1261569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1261922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1262245Z res = mod(**inputs) 2025-12-04T09:44:56.1262606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1263004Z outputs = self.model.decoder( 2025-12-04T09:44:56.1263407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1263799Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1264148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1264504Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1264871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1265223Z return func(*args, **kwargs) 2025-12-04T09:44:56.1265591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1265998Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1266379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1266734Z return func(*args, **kwargs) 2025-12-04T09:44:56.1267126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1267515Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1267644Z 2025-12-04T09:44:56.1267743Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1268088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1268398Z res = mod(**inputs) 2025-12-04T09:44:56.1268757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1269135Z outputs = self.model.decoder( 2025-12-04T09:44:56.1269511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1269921Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1270268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1270624Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1270994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1271357Z return func(*args, **kwargs) 2025-12-04T09:44:56.1271722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1272130Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1272697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1273065Z return func(*args, **kwargs) 2025-12-04T09:44:56.1273495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1273897Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1274039Z 2025-12-04T09:44:56.1274129Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1274359Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1274716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1275036Z res = mod(**inputs) 2025-12-04T09:44:56.1275410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1275805Z outputs = self.model.decoder( 2025-12-04T09:44:56.1276199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1276590Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1276926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1277281Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1277649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1278015Z return func(*args, **kwargs) 2025-12-04T09:44:56.1278378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1278790Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1279180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1279542Z return func(*args, **kwargs) 2025-12-04T09:44:56.1279907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1280320Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1280789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1281255Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1281441Z 2025-12-04T09:44:56.1281544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1281891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1282202Z res = mod(**inputs) 2025-12-04T09:44:56.1282559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1282946Z outputs = self.model.decoder( 2025-12-04T09:44:56.1283327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1283735Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1284101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1284460Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1284829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1285181Z return func(*args, **kwargs) 2025-12-04T09:44:56.1285554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1285972Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1286371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1286734Z return func(*args, **kwargs) 2025-12-04T09:44:56.1287118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1287558Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1287696Z 2025-12-04T09:44:56.1287800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1288154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1288476Z res = mod(**inputs) 2025-12-04T09:44:56.1288847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1289237Z outputs = self.model.decoder( 2025-12-04T09:44:56.1289628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1290029Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1290373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1290739Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1291164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1291653Z return func(*args, **kwargs) 2025-12-04T09:44:56.1292071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1292567Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1292766Z 2025-12-04T09:44:56.1292882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1293243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1293565Z res = mod(**inputs) 2025-12-04T09:44:56.1293943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1294349Z outputs = self.model.decoder( 2025-12-04T09:44:56.1294732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1295161Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1295512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1295873Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1296245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1296620Z return func(*args, **kwargs) 2025-12-04T09:44:56.1297002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1297439Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1297840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1298185Z return self.act(input) 2025-12-04T09:44:56.1298309Z 2025-12-04T09:44:56.1298425Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1298779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1299112Z res = mod(**inputs) 2025-12-04T09:44:56.1299492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1299916Z outputs = self.model.decoder( 2025-12-04T09:44:56.1300312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1300711Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1301056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1301439Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1301831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1302214Z return func(*args, **kwargs) 2025-12-04T09:44:56.1302607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1303009Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1303158Z 2025-12-04T09:44:56.1303265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1303631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1303961Z res = mod(**inputs) 2025-12-04T09:44:56.1304334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1304749Z outputs = self.model.decoder( 2025-12-04T09:44:56.1305152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1305562Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1305916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1306276Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1306658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1307029Z return func(*args, **kwargs) 2025-12-04T09:44:56.1307409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:44:56.1307810Z hidden_states = residual + hidden_states 2025-12-04T09:44:56.1307948Z 2025-12-04T09:44:56.1308055Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1308419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1308744Z res = mod(**inputs) 2025-12-04T09:44:56.1309135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1309519Z outputs = self.model.decoder( 2025-12-04T09:44:56.1309905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1310297Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1310632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1311000Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1311369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1311761Z return func(*args, **kwargs) 2025-12-04T09:44:56.1312157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1312575Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1312961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1313324Z return func(*args, **kwargs) 2025-12-04T09:44:56.1313690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1314150Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1314346Z 2025-12-04T09:44:56.1314457Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1314801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1315137Z res = mod(**inputs) 2025-12-04T09:44:56.1315503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1315898Z outputs = self.model.decoder( 2025-12-04T09:44:56.1316271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1316661Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1317003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1317357Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1317720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1318086Z return func(*args, **kwargs) 2025-12-04T09:44:56.1318456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1318872Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1319251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1319605Z return func(*args, **kwargs) 2025-12-04T09:44:56.1319970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1320344Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1320481Z 2025-12-04T09:44:56.1320579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1320917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1321214Z res = mod(**inputs) 2025-12-04T09:44:56.1321566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1321949Z outputs = self.model.decoder( 2025-12-04T09:44:56.1322341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1322715Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1323049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1323393Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1323753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1324102Z return func(*args, **kwargs) 2025-12-04T09:44:56.1324468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1324878Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1325300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1325698Z return func(*args, **kwargs) 2025-12-04T09:44:56.1326091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1326509Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1326650Z 2025-12-04T09:44:56.1326730Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1326973Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1327334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1327647Z res = mod(**inputs) 2025-12-04T09:44:56.1328019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1328420Z outputs = self.model.decoder( 2025-12-04T09:44:56.1328830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1329219Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1329571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1329939Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1330317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1330682Z return func(*args, **kwargs) 2025-12-04T09:44:56.1331080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1331650Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1332100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1332539Z return func(*args, **kwargs) 2025-12-04T09:44:56.1332954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1333434Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1333897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1334389Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1334575Z 2025-12-04T09:44:56.1334690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1335049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1335374Z res = mod(**inputs) 2025-12-04T09:44:56.1335747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1336197Z outputs = self.model.decoder( 2025-12-04T09:44:56.1336616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1337016Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1337366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1337729Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1338103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1338483Z return func(*args, **kwargs) 2025-12-04T09:44:56.1338872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1339286Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1339709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1340099Z return func(*args, **kwargs) 2025-12-04T09:44:56.1340483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1340882Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1341023Z 2025-12-04T09:44:56.1341126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1341480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1341804Z res = mod(**inputs) 2025-12-04T09:44:56.1342169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1342564Z outputs = self.model.decoder( 2025-12-04T09:44:56.1342954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1343379Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1343739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1344107Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1344488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1344857Z return func(*args, **kwargs) 2025-12-04T09:44:56.1345242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1345684Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1345855Z 2025-12-04T09:44:56.1345967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1346318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1346641Z res = mod(**inputs) 2025-12-04T09:44:56.1347020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1347413Z outputs = self.model.decoder( 2025-12-04T09:44:56.1347812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1348214Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1348566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1348923Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1349302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1349678Z return func(*args, **kwargs) 2025-12-04T09:44:56.1350060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1350501Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1350917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1351261Z return self.act(input) 2025-12-04T09:44:56.1351371Z 2025-12-04T09:44:56.1351471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1351824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1352152Z res = mod(**inputs) 2025-12-04T09:44:56.1352501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1352887Z outputs = self.model.decoder( 2025-12-04T09:44:56.1353263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1353673Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1354019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1354372Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1354736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1355095Z return func(*args, **kwargs) 2025-12-04T09:44:56.1355458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1355847Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1355977Z 2025-12-04T09:44:56.1356083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1356417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1356747Z res = mod(**inputs) 2025-12-04T09:44:56.1357115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1357505Z outputs = self.model.decoder( 2025-12-04T09:44:56.1357882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1358270Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1358613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1358962Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1359333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1359696Z return func(*args, **kwargs) 2025-12-04T09:44:56.1360071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1360481Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1360877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1361242Z return func(*args, **kwargs) 2025-12-04T09:44:56.1361612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1362068Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1362278Z 2025-12-04T09:44:56.1362379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1362736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1363047Z res = mod(**inputs) 2025-12-04T09:44:56.1363415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1363809Z outputs = self.model.decoder( 2025-12-04T09:44:56.1364206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1364589Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1364928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1365280Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1365649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1366006Z return func(*args, **kwargs) 2025-12-04T09:44:56.1366383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1366825Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1367217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1367577Z return func(*args, **kwargs) 2025-12-04T09:44:56.1367938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1368332Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1368462Z 2025-12-04T09:44:56.1368562Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1368912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1369229Z res = mod(**inputs) 2025-12-04T09:44:56.1369583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1369971Z outputs = self.model.decoder( 2025-12-04T09:44:56.1370371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1370755Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1371086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1371522Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1371894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1372258Z return func(*args, **kwargs) 2025-12-04T09:44:56.1372847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1373311Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1373740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1374103Z return func(*args, **kwargs) 2025-12-04T09:44:56.1374487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1374881Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1375015Z 2025-12-04T09:44:56.1375098Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1375322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1375670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1375990Z res = mod(**inputs) 2025-12-04T09:44:56.1376347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1376739Z outputs = self.model.decoder( 2025-12-04T09:44:56.1377118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1377523Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1377850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1378248Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1378615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1378967Z return func(*args, **kwargs) 2025-12-04T09:44:56.1379323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1379723Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1380106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1380463Z return func(*args, **kwargs) 2025-12-04T09:44:56.1380860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1381295Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1381730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1382197Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1382381Z 2025-12-04T09:44:56.1382484Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1382833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1383152Z res = mod(**inputs) 2025-12-04T09:44:56.1383506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1383896Z outputs = self.model.decoder( 2025-12-04T09:44:56.1384306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1384687Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1385029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1385384Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1385752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1386119Z return func(*args, **kwargs) 2025-12-04T09:44:56.1386506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1386911Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1387287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1387651Z return func(*args, **kwargs) 2025-12-04T09:44:56.1388021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1388415Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1388545Z 2025-12-04T09:44:56.1388645Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1388993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1389303Z res = mod(**inputs) 2025-12-04T09:44:56.1389659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1390044Z outputs = self.model.decoder( 2025-12-04T09:44:56.1390422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1390805Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1391135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1391487Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1391874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1392237Z return func(*args, **kwargs) 2025-12-04T09:44:56.1392603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1393034Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1393205Z 2025-12-04T09:44:56.1393317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1393661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1393975Z res = mod(**inputs) 2025-12-04T09:44:56.1394363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1394770Z outputs = self.model.decoder( 2025-12-04T09:44:56.1395150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1395542Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1395885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1396242Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1396606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1396975Z return func(*args, **kwargs) 2025-12-04T09:44:56.1397348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1397798Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1398178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1398512Z return self.act(input) 2025-12-04T09:44:56.1398616Z 2025-12-04T09:44:56.1398724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1399061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1399381Z res = mod(**inputs) 2025-12-04T09:44:56.1399727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1400092Z outputs = self.model.decoder( 2025-12-04T09:44:56.1400459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1400835Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1401166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1401501Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1401863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1402213Z return func(*args, **kwargs) 2025-12-04T09:44:56.1402573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1402948Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1403081Z 2025-12-04T09:44:56.1403176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1403510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1403804Z res = mod(**inputs) 2025-12-04T09:44:56.1404154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1404534Z outputs = self.model.decoder( 2025-12-04T09:44:56.1404928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1405308Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1405650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1405995Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1406348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1406738Z return func(*args, **kwargs) 2025-12-04T09:44:56.1407111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:44:56.1407521Z hidden_states = residual + hidden_states 2025-12-04T09:44:56.1407650Z 2025-12-04T09:44:56.1407751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1408119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1408434Z res = mod(**inputs) 2025-12-04T09:44:56.1408804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1409192Z outputs = self.model.decoder( 2025-12-04T09:44:56.1409581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1409976Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1410319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1410682Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1411082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1411536Z return func(*args, **kwargs) 2025-12-04T09:44:56.1411921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1412372Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1412797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1413195Z return func(*args, **kwargs) 2025-12-04T09:44:56.1413590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1414074Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1414272Z 2025-12-04T09:44:56.1414381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1414719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1415028Z res = mod(**inputs) 2025-12-04T09:44:56.1415392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1415779Z outputs = self.model.decoder( 2025-12-04T09:44:56.1416150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1416532Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1416872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1417215Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1417580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1417951Z return func(*args, **kwargs) 2025-12-04T09:44:56.1418321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1418754Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1419146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1419514Z return func(*args, **kwargs) 2025-12-04T09:44:56.1419887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1420273Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1420408Z 2025-12-04T09:44:56.1420507Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1420853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1421176Z res = mod(**inputs) 2025-12-04T09:44:56.1421541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1421954Z outputs = self.model.decoder( 2025-12-04T09:44:56.1422344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1422727Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1423073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1423429Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1423796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1424167Z return func(*args, **kwargs) 2025-12-04T09:44:56.1424546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1424979Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1425367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1425733Z return func(*args, **kwargs) 2025-12-04T09:44:56.1426108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1426506Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1426644Z 2025-12-04T09:44:56.1426726Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1426958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1427309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1427619Z res = mod(**inputs) 2025-12-04T09:44:56.1427986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1428380Z outputs = self.model.decoder( 2025-12-04T09:44:56.1428763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1429145Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1429489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1429847Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1430210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1430576Z return func(*args, **kwargs) 2025-12-04T09:44:56.1430951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1431371Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1431759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1432129Z return func(*args, **kwargs) 2025-12-04T09:44:56.1432519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1432999Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1433426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1433896Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1433900Z 2025-12-04T09:44:56.1433999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1434200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1435090Z res = mod(**inputs) 2025-12-04T09:44:56.1435361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1435455Z outputs = self.model.decoder( 2025-12-04T09:44:56.1435731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1435813Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1436038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1436118Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1436377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1436444Z return func(*args, **kwargs) 2025-12-04T09:44:56.1436703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1436822Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1437060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1437138Z return func(*args, **kwargs) 2025-12-04T09:44:56.1437394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1437484Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1437487Z 2025-12-04T09:44:56.1437590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1437780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1437848Z res = mod(**inputs) 2025-12-04T09:44:56.1438104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1438177Z outputs = self.model.decoder( 2025-12-04T09:44:56.1438440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1438509Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1438728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1438804Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1439036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1439108Z return func(*args, **kwargs) 2025-12-04T09:44:56.1439359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1439476Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1439486Z 2025-12-04T09:44:56.1439588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1439779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1439846Z res = mod(**inputs) 2025-12-04T09:44:56.1440120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1440190Z outputs = self.model.decoder( 2025-12-04T09:44:56.1440451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1440519Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1440737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1440810Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1441043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1441135Z return func(*args, **kwargs) 2025-12-04T09:44:56.1441412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1441530Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1441749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1441820Z return self.act(input) 2025-12-04T09:44:56.1441824Z 2025-12-04T09:44:56.1441933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1442132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1442196Z res = mod(**inputs) 2025-12-04T09:44:56.1442470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1442561Z outputs = self.model.decoder( 2025-12-04T09:44:56.1442835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1442906Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1443129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1443215Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1443459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1443526Z return func(*args, **kwargs) 2025-12-04T09:44:56.1443797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1443878Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1443882Z 2025-12-04T09:44:56.1443990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1444192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1444255Z res = mod(**inputs) 2025-12-04T09:44:56.1444530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1444601Z outputs = self.model.decoder( 2025-12-04T09:44:56.1444867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1444946Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1445169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1445254Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1445498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1445571Z return func(*args, **kwargs) 2025-12-04T09:44:56.1445846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1445961Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1446208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1446278Z return func(*args, **kwargs) 2025-12-04T09:44:56.1446537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1446694Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1446698Z 2025-12-04T09:44:56.1446800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1446995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1447085Z res = mod(**inputs) 2025-12-04T09:44:56.1447388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1447470Z outputs = self.model.decoder( 2025-12-04T09:44:56.1447739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1447809Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1448034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1448112Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1448361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1448428Z return func(*args, **kwargs) 2025-12-04T09:44:56.1448687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1448820Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1449065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1449130Z return func(*args, **kwargs) 2025-12-04T09:44:56.1449395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1449474Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1449478Z 2025-12-04T09:44:56.1449585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1449777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1449841Z res = mod(**inputs) 2025-12-04T09:44:56.1450107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1450182Z outputs = self.model.decoder( 2025-12-04T09:44:56.1450456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1450532Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1450764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1450853Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1451104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1451174Z return func(*args, **kwargs) 2025-12-04T09:44:56.1451550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1451661Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1451929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1452003Z return func(*args, **kwargs) 2025-12-04T09:44:56.1452299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1452400Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1452405Z 2025-12-04T09:44:56.1452493Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1452600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1452815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1452881Z res = mod(**inputs) 2025-12-04T09:44:56.1453164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1453236Z outputs = self.model.decoder( 2025-12-04T09:44:56.1453520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1453620Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1453842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1453929Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1454167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1454234Z return func(*args, **kwargs) 2025-12-04T09:44:56.1454502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1454597Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1454834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1454931Z return func(*args, **kwargs) 2025-12-04T09:44:56.1455196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1455301Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1455593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1455727Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1455730Z 2025-12-04T09:44:56.1455840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1456041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1456112Z res = mod(**inputs) 2025-12-04T09:44:56.1456378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1456452Z outputs = self.model.decoder( 2025-12-04T09:44:56.1456725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1456799Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1457021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1457108Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1457354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1457432Z return func(*args, **kwargs) 2025-12-04T09:44:56.1457693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1457790Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1458042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1458110Z return func(*args, **kwargs) 2025-12-04T09:44:56.1458404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1458488Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1458492Z 2025-12-04T09:44:56.1458593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1458794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1458855Z res = mod(**inputs) 2025-12-04T09:44:56.1459116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1459196Z outputs = self.model.decoder( 2025-12-04T09:44:56.1459456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1459562Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1459808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1459888Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1460136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1460208Z return func(*args, **kwargs) 2025-12-04T09:44:56.1460472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1460599Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1460603Z 2025-12-04T09:44:56.1460706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1460912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1461000Z res = mod(**inputs) 2025-12-04T09:44:56.1461276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1461356Z outputs = self.model.decoder( 2025-12-04T09:44:56.1461623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1461702Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1461927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1462005Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1462256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1462323Z return func(*args, **kwargs) 2025-12-04T09:44:56.1462588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1462716Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1462934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1463009Z return self.act(input) 2025-12-04T09:44:56.1463013Z 2025-12-04T09:44:56.1463116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1463318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1463390Z res = mod(**inputs) 2025-12-04T09:44:56.1463656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1463733Z outputs = self.model.decoder( 2025-12-04T09:44:56.1464054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1464128Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1464360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1464455Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1464700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1464776Z return func(*args, **kwargs) 2025-12-04T09:44:56.1465035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1465122Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1465125Z 2025-12-04T09:44:56.1465226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1465420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1465511Z res = mod(**inputs) 2025-12-04T09:44:56.1465793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1465866Z outputs = self.model.decoder( 2025-12-04T09:44:56.1466143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1466215Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1466446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1466524Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1466769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1466844Z return func(*args, **kwargs) 2025-12-04T09:44:56.1467113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:44:56.1467225Z hidden_states = residual + hidden_states 2025-12-04T09:44:56.1467229Z 2025-12-04T09:44:56.1467334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1467534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1467605Z res = mod(**inputs) 2025-12-04T09:44:56.1467872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1467941Z outputs = self.model.decoder( 2025-12-04T09:44:56.1468271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1468341Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1468566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1468644Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1468887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1468964Z return func(*args, **kwargs) 2025-12-04T09:44:56.1469227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1469329Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1469569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1469641Z return func(*args, **kwargs) 2025-12-04T09:44:56.1469907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1470056Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1470063Z 2025-12-04T09:44:56.1470163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1470365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1470445Z res = mod(**inputs) 2025-12-04T09:44:56.1470719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1470791Z outputs = self.model.decoder( 2025-12-04T09:44:56.1471054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1471131Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1471352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1471438Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1471695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1471764Z return func(*args, **kwargs) 2025-12-04T09:44:56.1472059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1472160Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1472570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1472656Z return func(*args, **kwargs) 2025-12-04T09:44:56.1472916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1473003Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1473007Z 2025-12-04T09:44:56.1473110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1473312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1473431Z res = mod(**inputs) 2025-12-04T09:44:56.1473698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1473770Z outputs = self.model.decoder( 2025-12-04T09:44:56.1474051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1474121Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1474345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1474421Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1474655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1474731Z return func(*args, **kwargs) 2025-12-04T09:44:56.1474985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1475091Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1475330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1475397Z return func(*args, **kwargs) 2025-12-04T09:44:56.1475658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1475743Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1475747Z 2025-12-04T09:44:56.1475827Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1475936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1476126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1476199Z res = mod(**inputs) 2025-12-04T09:44:56.1476455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1476527Z outputs = self.model.decoder( 2025-12-04T09:44:56.1476817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1476890Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1477119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1477194Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1477429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1477502Z return func(*args, **kwargs) 2025-12-04T09:44:56.1477756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1477875Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1478145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1478214Z return func(*args, **kwargs) 2025-12-04T09:44:56.1478482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1478578Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1478866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1479006Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1479010Z 2025-12-04T09:44:56.1479111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1479310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1479393Z res = mod(**inputs) 2025-12-04T09:44:56.1479661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1479738Z outputs = self.model.decoder( 2025-12-04T09:44:56.1479998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1480067Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1480296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1480371Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1480620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1480687Z return func(*args, **kwargs) 2025-12-04T09:44:56.1480952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1481058Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1481301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1481367Z return func(*args, **kwargs) 2025-12-04T09:44:56.1481636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1481715Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1481719Z 2025-12-04T09:44:56.1481827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1482022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1482085Z res = mod(**inputs) 2025-12-04T09:44:56.1482359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1482433Z outputs = self.model.decoder( 2025-12-04T09:44:56.1482720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1482792Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1483005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1483089Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1483324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1483391Z return func(*args, **kwargs) 2025-12-04T09:44:56.1483652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1483783Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1483788Z 2025-12-04T09:44:56.1483895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1484103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1484166Z res = mod(**inputs) 2025-12-04T09:44:56.1484427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1484495Z outputs = self.model.decoder( 2025-12-04T09:44:56.1484758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1484824Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1485035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1485115Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1485369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1485436Z return func(*args, **kwargs) 2025-12-04T09:44:56.1485698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1485812Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1486026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1486092Z return self.act(input) 2025-12-04T09:44:56.1486095Z 2025-12-04T09:44:56.1486191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1486389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1486449Z res = mod(**inputs) 2025-12-04T09:44:56.1486705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1486785Z outputs = self.model.decoder( 2025-12-04T09:44:56.1487042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1487118Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1487333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1487408Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1487649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1487715Z return func(*args, **kwargs) 2025-12-04T09:44:56.1487974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1488056Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1488061Z 2025-12-04T09:44:56.1488159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1488355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1488430Z res = mod(**inputs) 2025-12-04T09:44:56.1488688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1488766Z outputs = self.model.decoder( 2025-12-04T09:44:56.1489028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1489107Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1489329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1489406Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1489671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1489740Z return func(*args, **kwargs) 2025-12-04T09:44:56.1490023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1490125Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1490371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1490447Z return func(*args, **kwargs) 2025-12-04T09:44:56.1490710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1490860Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1490864Z 2025-12-04T09:44:56.1491382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1491649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1491776Z res = mod(**inputs) 2025-12-04T09:44:56.1492091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1492197Z outputs = self.model.decoder( 2025-12-04T09:44:56.1492586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1492695Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1492946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1493077Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1493344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1493462Z return func(*args, **kwargs) 2025-12-04T09:44:56.1493769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1493903Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1494198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1494288Z return func(*args, **kwargs) 2025-12-04T09:44:56.1494608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1494700Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1494705Z 2025-12-04T09:44:56.1494848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1495105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1495191Z res = mod(**inputs) 2025-12-04T09:44:56.1495519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1495625Z outputs = self.model.decoder( 2025-12-04T09:44:56.1495918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1496069Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1496310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1523535Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1524020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1524106Z return func(*args, **kwargs) 2025-12-04T09:44:56.1524410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1524664Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1524960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1525050Z return func(*args, **kwargs) 2025-12-04T09:44:56.1525328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1525432Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1525438Z 2025-12-04T09:44:56.1525525Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1525642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1525867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1525938Z res = mod(**inputs) 2025-12-04T09:44:56.1526223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1526340Z outputs = self.model.decoder( 2025-12-04T09:44:56.1526620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1526703Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1526939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1527021Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1527284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1527354Z return func(*args, **kwargs) 2025-12-04T09:44:56.1527640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1527742Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1527996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1528078Z return func(*args, **kwargs) 2025-12-04T09:44:56.1528354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1528457Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1528768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1528907Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1528912Z 2025-12-04T09:44:56.1529031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1529241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1529308Z res = mod(**inputs) 2025-12-04T09:44:56.1529597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1529677Z outputs = self.model.decoder( 2025-12-04T09:44:56.1530006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1530083Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1530308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1530395Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1530643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1530711Z return func(*args, **kwargs) 2025-12-04T09:44:56.1530984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1531112Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1531475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1531552Z return func(*args, **kwargs) 2025-12-04T09:44:56.1531863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1531961Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1531965Z 2025-12-04T09:44:56.1532083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1532318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1532390Z res = mod(**inputs) 2025-12-04T09:44:56.1532676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1532782Z outputs = self.model.decoder( 2025-12-04T09:44:56.1533066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1533142Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1533388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1533467Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1533718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1533786Z return func(*args, **kwargs) 2025-12-04T09:44:56.1534053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1534184Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1534187Z 2025-12-04T09:44:56.1534294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1534505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1534572Z res = mod(**inputs) 2025-12-04T09:44:56.1534840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1534923Z outputs = self.model.decoder( 2025-12-04T09:44:56.1535187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1535257Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1535493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1535572Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1535825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1535897Z return func(*args, **kwargs) 2025-12-04T09:44:56.1536164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1536311Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1536527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1536599Z return self.act(input) 2025-12-04T09:44:56.1536611Z 2025-12-04T09:44:56.1536714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1536910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1536979Z res = mod(**inputs) 2025-12-04T09:44:56.1537246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1537340Z outputs = self.model.decoder( 2025-12-04T09:44:56.1537616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1537702Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1537933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1538011Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1538282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1538358Z return func(*args, **kwargs) 2025-12-04T09:44:56.1538615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1538695Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1538706Z 2025-12-04T09:44:56.1538807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1539019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1539088Z res = mod(**inputs) 2025-12-04T09:44:56.1539346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1539417Z outputs = self.model.decoder( 2025-12-04T09:44:56.1539681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1539749Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1539970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1540046Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1540280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1540357Z return func(*args, **kwargs) 2025-12-04T09:44:56.1540612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T09:44:56.1540689Z hidden_states = residual + hidden_states 2025-12-04T09:44:56.1540693Z 2025-12-04T09:44:56.1540801Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1540990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1541058Z res = mod(**inputs) 2025-12-04T09:44:56.1541312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1541381Z outputs = self.model.decoder( 2025-12-04T09:44:56.1541646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1541715Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1541930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1542013Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1542267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1542343Z return func(*args, **kwargs) 2025-12-04T09:44:56.1542598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1542695Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1542939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1543005Z return func(*args, **kwargs) 2025-12-04T09:44:56.1543265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T09:44:56.1543435Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T09:44:56.1543439Z 2025-12-04T09:44:56.1543553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1543758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1543822Z res = mod(**inputs) 2025-12-04T09:44:56.1544080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1544162Z outputs = self.model.decoder( 2025-12-04T09:44:56.1544418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1544495Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1544709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1544805Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1545052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1545119Z return func(*args, **kwargs) 2025-12-04T09:44:56.1545380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1545475Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1545709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1545781Z return func(*args, **kwargs) 2025-12-04T09:44:56.1546036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T09:44:56.1546112Z key_states = self.k_proj(current_states) 2025-12-04T09:44:56.1546125Z 2025-12-04T09:44:56.1546224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1546412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1546481Z res = mod(**inputs) 2025-12-04T09:44:56.1546735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1546804Z outputs = self.model.decoder( 2025-12-04T09:44:56.1547064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1547132Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1547353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1547428Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1547657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1547731Z return func(*args, **kwargs) 2025-12-04T09:44:56.1547984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1548093Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1548338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1548404Z return func(*args, **kwargs) 2025-12-04T09:44:56.1548668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T09:44:56.1548751Z value_states = self.v_proj(current_states) 2025-12-04T09:44:56.1548755Z 2025-12-04T09:44:56.1548835Z cudagraph partition due to non gpu ops 2025-12-04T09:44:56.1548942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1549151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1549216Z res = mod(**inputs) 2025-12-04T09:44:56.1549500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1549572Z outputs = self.model.decoder( 2025-12-04T09:44:56.1549838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1549906Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1550128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1550202Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1550434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1550505Z return func(*args, **kwargs) 2025-12-04T09:44:56.1550790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1550901Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1551131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1551193Z return func(*args, **kwargs) 2025-12-04T09:44:56.1551450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T09:44:56.1551542Z attn_output, attn_weights = attention_interface( 2025-12-04T09:44:56.1551825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T09:44:56.1551954Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:44:56.1551959Z 2025-12-04T09:44:56.1552057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1552251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1552311Z res = mod(**inputs) 2025-12-04T09:44:56.1552562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1552638Z outputs = self.model.decoder( 2025-12-04T09:44:56.1552887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1552961Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1553170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1553243Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1553479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1553545Z return func(*args, **kwargs) 2025-12-04T09:44:56.1553801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T09:44:56.1553908Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:44:56.1554142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1554212Z return func(*args, **kwargs) 2025-12-04T09:44:56.1554461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T09:44:56.1554538Z attn_output = self.out_proj(attn_output) 2025-12-04T09:44:56.1554541Z 2025-12-04T09:44:56.1554645Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1554834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1554920Z res = mod(**inputs) 2025-12-04T09:44:56.1555196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1555268Z outputs = self.model.decoder( 2025-12-04T09:44:56.1555520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1555586Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1555796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1555875Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1556103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1556173Z return func(*args, **kwargs) 2025-12-04T09:44:56.1556427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1556561Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1556566Z 2025-12-04T09:44:56.1556675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1556865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1556934Z res = mod(**inputs) 2025-12-04T09:44:56.1557188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1557257Z outputs = self.model.decoder( 2025-12-04T09:44:56.1557519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1557586Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1557806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1557890Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1558331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1558412Z return func(*args, **kwargs) 2025-12-04T09:44:56.1558660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T09:44:56.1558771Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:44:56.1558983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:44:56.1559051Z return self.act(input) 2025-12-04T09:44:56.1559055Z 2025-12-04T09:44:56.1559161Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1559345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1559410Z res = mod(**inputs) 2025-12-04T09:44:56.1559670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T09:44:56.1559739Z outputs = self.model.decoder( 2025-12-04T09:44:56.1560008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T09:44:56.1560086Z layer_outputs = decoder_layer( 2025-12-04T09:44:56.1560296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:44:56.1560382Z return super().__call__(*args, **kwargs) 2025-12-04T09:44:56.1560609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:44:56.1560674Z return func(*args, **kwargs) 2025-12-04T09:44:56.1560929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T09:44:56.1561026Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:44:56.1561030Z 2025-12-04T09:44:56.1561152Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1561340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1561400Z res = mod(**inputs) 2025-12-04T09:44:56.1561654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1649, in forward 2025-12-04T09:44:56.1561727Z logits = self.lm_head(outputs[0]) 2025-12-04T09:44:56.1561731Z 2025-12-04T09:44:56.1561825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:44:56.1562018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:44:56.1562076Z res = mod(**inputs) 2025-12-04T09:44:56.1562330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1655, in forward 2025-12-04T09:44:56.1562490Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:44:56.1562495Z 2025-12-04T09:45:06.3783051Z Compilation time (from dynamo_timed): 16.661859976 2025-12-04T09:45:06.3804948Z pass 2025-12-04T09:45:06.3805329Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:06.3806210Z TIMING: _recursive_pre_grad_passes:0.00735 _recursive_joint_graph_passes:0.64068 _recursive_post_grad_passes:0.06642 async_compile.wait:0.90296 code_gen:9.99087 inductor_compile:11.28682 backend_compile:14.31678 gc:0.00092 entire_frame_compile:16.66186 total_wall_time:16.66186 2025-12-04T09:45:06.3807211Z STATS: call_* op count: 333 | FakeTensorMode.__torch_dispatch__:7342 | FakeTensor.__torch_dispatch__:4382 | ProxyTorchDispatchMode.__torch_dispatch__:2031 2025-12-04T09:45:06.3807741Z Dynamo produced 1 graphs covering 333 ops with 0 graph breaks (0 unique) 2025-12-04T09:45:08.9282876Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:45:08.9283867Z import pynvml # type: ignore[import] 2025-12-04T09:45:12.3858262Z 2025-12-04T09:45:12.3973282Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-12-04T09:45:12.3973965Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-12-04T09:45:13.4525346Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-12-04T09:45:13.4526758Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-12-04T09:45:13.4528679Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-12-04T09:45:13.4530133Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-12-04T09:45:13.6210649Z 2025-12-04T09:45:13.6211800Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:45:13.6231889Z cpu eval RobertaForCausalLM 2025-12-04T09:45:14.2050550Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:14.4839074Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:14.7633339Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:22.0876653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0877208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0877568Z res = mod(**inputs) 2025-12-04T09:45:22.0878012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0878464Z outputs = self.roberta( 2025-12-04T09:45:22.0878953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T09:45:22.0879408Z embedding_output = self.embeddings( 2025-12-04T09:45:22.0879849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T09:45:22.0880502Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T09:45:22.0881139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in create_position_ids_from_input_ids 2025-12-04T09:45:22.0881656Z mask = input_ids.ne(padding_idx).int() 2025-12-04T09:45:22.0881814Z 2025-12-04T09:45:22.0881903Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0882126Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0882339Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0882553Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0882766Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0882972Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0883188Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0883401Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0883619Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0883824Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0884043Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0884260Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0884503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0884902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0885258Z res = mod(**inputs) 2025-12-04T09:45:22.0885653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0886128Z outputs = self.roberta( 2025-12-04T09:45:22.0886549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T09:45:22.0886997Z embedding_output = self.embeddings( 2025-12-04T09:45:22.0887422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T09:45:22.0887993Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T09:45:22.0888786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in create_position_ids_from_input_ids 2025-12-04T09:45:22.0889467Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:45:22.0889729Z 2025-12-04T09:45:22.0889846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0890260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0890626Z res = mod(**inputs) 2025-12-04T09:45:22.0891030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0891684Z outputs = self.roberta( 2025-12-04T09:45:22.0892118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T09:45:22.0892670Z embedding_output = self.embeddings( 2025-12-04T09:45:22.0893105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T09:45:22.0893671Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T09:45:22.0894308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in create_position_ids_from_input_ids 2025-12-04T09:45:22.0894924Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T09:45:22.0895172Z 2025-12-04T09:45:22.0895277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0895642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0895994Z res = mod(**inputs) 2025-12-04T09:45:22.0896365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0896755Z outputs = self.roberta( 2025-12-04T09:45:22.0897132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0897533Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0897938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0898359Z layer_outputs = layer_module( 2025-12-04T09:45:22.0898740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0899137Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0899577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0899984Z return func(*args, **kwargs) 2025-12-04T09:45:22.0900397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0900829Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0901235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0901638Z return func(*args, **kwargs) 2025-12-04T09:45:22.0902051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.0902466Z self_outputs = self.self( 2025-12-04T09:45:22.0902848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0903225Z return func(*args, **kwargs) 2025-12-04T09:45:22.0903611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.0904186Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.0904466Z 2025-12-04T09:45:22.0904578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0904965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0905306Z res = mod(**inputs) 2025-12-04T09:45:22.0905685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0906096Z outputs = self.roberta( 2025-12-04T09:45:22.0906473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0906910Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0907338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0907757Z layer_outputs = layer_module( 2025-12-04T09:45:22.0908131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0908512Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0908914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0909313Z return func(*args, **kwargs) 2025-12-04T09:45:22.0909717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0910140Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0910545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0910961Z return func(*args, **kwargs) 2025-12-04T09:45:22.0911372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.0911780Z self_outputs = self.self( 2025-12-04T09:45:22.0912162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0912557Z return func(*args, **kwargs) 2025-12-04T09:45:22.0912951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.0913364Z self.key(current_states) 2025-12-04T09:45:22.0913492Z 2025-12-04T09:45:22.0913601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0913980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0914319Z res = mod(**inputs) 2025-12-04T09:45:22.0914712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0915132Z outputs = self.roberta( 2025-12-04T09:45:22.0915521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0915940Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0916353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0916769Z layer_outputs = layer_module( 2025-12-04T09:45:22.0917147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0917537Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0917935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0918329Z return func(*args, **kwargs) 2025-12-04T09:45:22.0918745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0919180Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0919584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0919981Z return func(*args, **kwargs) 2025-12-04T09:45:22.0920385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.0920803Z self_outputs = self.self( 2025-12-04T09:45:22.0921188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0921606Z return func(*args, **kwargs) 2025-12-04T09:45:22.0922012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.0922450Z self.value(current_states) 2025-12-04T09:45:22.0922581Z 2025-12-04T09:45:22.0922667Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0922920Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0923305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0923648Z res = mod(**inputs) 2025-12-04T09:45:22.0924035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0924465Z outputs = self.roberta( 2025-12-04T09:45:22.0924879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0925308Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0925751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0926179Z layer_outputs = layer_module( 2025-12-04T09:45:22.0926561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0926955Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0927681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0928097Z return func(*args, **kwargs) 2025-12-04T09:45:22.0928512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0928952Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0929375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0929789Z return func(*args, **kwargs) 2025-12-04T09:45:22.0930208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.0930635Z self_outputs = self.self( 2025-12-04T09:45:22.0931034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0931580Z return func(*args, **kwargs) 2025-12-04T09:45:22.0931999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.0932502Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.0932724Z 2025-12-04T09:45:22.0932845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0933244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0933593Z res = mod(**inputs) 2025-12-04T09:45:22.0934003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0934475Z outputs = self.roberta( 2025-12-04T09:45:22.0934904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0935325Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0935749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0936171Z layer_outputs = layer_module( 2025-12-04T09:45:22.0936542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0936934Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0937345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0937773Z return func(*args, **kwargs) 2025-12-04T09:45:22.0938200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0938642Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0939060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0939587Z return func(*args, **kwargs) 2025-12-04T09:45:22.0940004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.0940497Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.0940965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.0941419Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.0941575Z 2025-12-04T09:45:22.0941686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0942074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0942416Z res = mod(**inputs) 2025-12-04T09:45:22.0942792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0943200Z outputs = self.roberta( 2025-12-04T09:45:22.0943591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0943996Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0944403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0944815Z layer_outputs = layer_module( 2025-12-04T09:45:22.0945186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0945566Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0945965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0946356Z return func(*args, **kwargs) 2025-12-04T09:45:22.0946746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.0947178Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.0947612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.0948022Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.0948469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.0948976Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.0949476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.0949909Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.0950054Z 2025-12-04T09:45:22.0950165Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0950546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0950884Z res = mod(**inputs) 2025-12-04T09:45:22.0951267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0951682Z outputs = self.roberta( 2025-12-04T09:45:22.0952082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0952521Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0952941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0953359Z layer_outputs = layer_module( 2025-12-04T09:45:22.0953724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0954088Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0954460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0954836Z return func(*args, **kwargs) 2025-12-04T09:45:22.0955216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.0955622Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.0956029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.0956442Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.0956869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.0957335Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.0957773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.0958203Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.0958584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.0958980Z return self.act(input) 2025-12-04T09:45:22.0959098Z 2025-12-04T09:45:22.0959202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0959567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0959882Z res = mod(**inputs) 2025-12-04T09:45:22.0960260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0960649Z outputs = self.roberta( 2025-12-04T09:45:22.0961020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0961403Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0961790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0962179Z layer_outputs = layer_module( 2025-12-04T09:45:22.0962518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0962882Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0963265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0963638Z return func(*args, **kwargs) 2025-12-04T09:45:22.0964029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.0964436Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.0964838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.0965232Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.0965650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.0966136Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.0966584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.0967000Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.0967158Z 2025-12-04T09:45:22.0967265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0967622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0967950Z res = mod(**inputs) 2025-12-04T09:45:22.0968303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0968691Z outputs = self.roberta( 2025-12-04T09:45:22.0969061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0969450Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0969829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0970243Z layer_outputs = layer_module( 2025-12-04T09:45:22.0970591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0970953Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0971459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0971881Z return func(*args, **kwargs) 2025-12-04T09:45:22.0972493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0972925Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0973343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0973752Z return func(*args, **kwargs) 2025-12-04T09:45:22.0974132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.0974522Z self_outputs = self.self( 2025-12-04T09:45:22.0974905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0975308Z return func(*args, **kwargs) 2025-12-04T09:45:22.0975703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.0976269Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.0976556Z 2025-12-04T09:45:22.0976669Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0977052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0977379Z res = mod(**inputs) 2025-12-04T09:45:22.0977769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0978175Z outputs = self.roberta( 2025-12-04T09:45:22.0978627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0979044Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0979453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0979866Z layer_outputs = layer_module( 2025-12-04T09:45:22.0980228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0980609Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0981021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0981460Z return func(*args, **kwargs) 2025-12-04T09:45:22.0981892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0982286Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0982658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0983014Z return func(*args, **kwargs) 2025-12-04T09:45:22.0983386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.0983770Z self_outputs = self.self( 2025-12-04T09:45:22.0984121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0984477Z return func(*args, **kwargs) 2025-12-04T09:45:22.0984847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.0985282Z self.key(current_states) 2025-12-04T09:45:22.0985392Z 2025-12-04T09:45:22.0985493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0985842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0986152Z res = mod(**inputs) 2025-12-04T09:45:22.0986505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0986870Z outputs = self.roberta( 2025-12-04T09:45:22.0987230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0987608Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0987974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0988354Z layer_outputs = layer_module( 2025-12-04T09:45:22.0988694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0989051Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0989412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0989774Z return func(*args, **kwargs) 2025-12-04T09:45:22.0990142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0990529Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0990895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0991254Z return func(*args, **kwargs) 2025-12-04T09:45:22.0991621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.0991993Z self_outputs = self.self( 2025-12-04T09:45:22.0992359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0992715Z return func(*args, **kwargs) 2025-12-04T09:45:22.0993089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.0993471Z self.value(current_states) 2025-12-04T09:45:22.0993591Z 2025-12-04T09:45:22.0993672Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.0993910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.0994258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.0994577Z res = mod(**inputs) 2025-12-04T09:45:22.0994942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.0995335Z outputs = self.roberta( 2025-12-04T09:45:22.0995710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.0996093Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.0996468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.0996847Z layer_outputs = layer_module( 2025-12-04T09:45:22.0997181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.0997536Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.0997912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0998277Z return func(*args, **kwargs) 2025-12-04T09:45:22.0998683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.0999092Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.0999515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.0999882Z return func(*args, **kwargs) 2025-12-04T09:45:22.1000264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1000658Z self_outputs = self.self( 2025-12-04T09:45:22.1001013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1001386Z return func(*args, **kwargs) 2025-12-04T09:45:22.1001764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1002223Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1002407Z 2025-12-04T09:45:22.1002512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1002872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1003194Z res = mod(**inputs) 2025-12-04T09:45:22.1003558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1003936Z outputs = self.roberta( 2025-12-04T09:45:22.1004309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1004696Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1005076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1005470Z layer_outputs = layer_module( 2025-12-04T09:45:22.1005818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1006180Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1006568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1006939Z return func(*args, **kwargs) 2025-12-04T09:45:22.1007320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1007715Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1008099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1008470Z return func(*args, **kwargs) 2025-12-04T09:45:22.1008849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1009310Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1009778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1010187Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1010325Z 2025-12-04T09:45:22.1010438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1010805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1011143Z res = mod(**inputs) 2025-12-04T09:45:22.1011637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1012047Z outputs = self.roberta( 2025-12-04T09:45:22.1012444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1012901Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1013325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1013744Z layer_outputs = layer_module( 2025-12-04T09:45:22.1014127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1014527Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1014939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1015321Z return func(*args, **kwargs) 2025-12-04T09:45:22.1015710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1016122Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1016552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1016979Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1017445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1017961Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1018434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1018871Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1019021Z 2025-12-04T09:45:22.1019145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1019538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1019893Z res = mod(**inputs) 2025-12-04T09:45:22.1020290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1020713Z outputs = self.roberta( 2025-12-04T09:45:22.1021133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1021555Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1021963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1022428Z layer_outputs = layer_module( 2025-12-04T09:45:22.1022787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1023170Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1023572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1023993Z return func(*args, **kwargs) 2025-12-04T09:45:22.1024418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1024831Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1025207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1025568Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1025976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1026429Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1026847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1027250Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1027641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1027972Z return self.act(input) 2025-12-04T09:45:22.1028077Z 2025-12-04T09:45:22.1028176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1028518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1028830Z res = mod(**inputs) 2025-12-04T09:45:22.1029180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1029540Z outputs = self.roberta( 2025-12-04T09:45:22.1029894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1030270Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1030629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1031003Z layer_outputs = layer_module( 2025-12-04T09:45:22.1031338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1031680Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1032032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1032382Z return func(*args, **kwargs) 2025-12-04T09:45:22.1032744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1033129Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1033502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1033873Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1034283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1034761Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1035188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1035574Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1035705Z 2025-12-04T09:45:22.1035811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1036144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1036455Z res = mod(**inputs) 2025-12-04T09:45:22.1036813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1037217Z outputs = self.roberta( 2025-12-04T09:45:22.1037577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1037992Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1038377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1038752Z layer_outputs = layer_module( 2025-12-04T09:45:22.1039092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1039456Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1039823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1040179Z return func(*args, **kwargs) 2025-12-04T09:45:22.1040560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1040966Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1041336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1041696Z return func(*args, **kwargs) 2025-12-04T09:45:22.1042068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1042449Z self_outputs = self.self( 2025-12-04T09:45:22.1042795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1043155Z return func(*args, **kwargs) 2025-12-04T09:45:22.1043526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1044044Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1044302Z 2025-12-04T09:45:22.1044404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1044759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1045077Z res = mod(**inputs) 2025-12-04T09:45:22.1045430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1045805Z outputs = self.roberta( 2025-12-04T09:45:22.1046169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1046550Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1046919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1047299Z layer_outputs = layer_module( 2025-12-04T09:45:22.1047643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1047993Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1048377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1048740Z return func(*args, **kwargs) 2025-12-04T09:45:22.1049107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1049490Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1049868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1050230Z return func(*args, **kwargs) 2025-12-04T09:45:22.1050599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1050999Z self_outputs = self.self( 2025-12-04T09:45:22.1051487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1051933Z return func(*args, **kwargs) 2025-12-04T09:45:22.1052345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1052752Z self.key(current_states) 2025-12-04T09:45:22.1052873Z 2025-12-04T09:45:22.1052977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1053341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1053654Z res = mod(**inputs) 2025-12-04T09:45:22.1054024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1054405Z outputs = self.roberta( 2025-12-04T09:45:22.1054773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1055199Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1055602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1055986Z layer_outputs = layer_module( 2025-12-04T09:45:22.1056327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1056689Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1057067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1057436Z return func(*args, **kwargs) 2025-12-04T09:45:22.1057804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1058208Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1058589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1058949Z return func(*args, **kwargs) 2025-12-04T09:45:22.1059332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1059720Z self_outputs = self.self( 2025-12-04T09:45:22.1060079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1060444Z return func(*args, **kwargs) 2025-12-04T09:45:22.1060824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1061216Z self.value(current_states) 2025-12-04T09:45:22.1061333Z 2025-12-04T09:45:22.1061416Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1061662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1062024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1062348Z res = mod(**inputs) 2025-12-04T09:45:22.1062730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1063112Z outputs = self.roberta( 2025-12-04T09:45:22.1063476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1063849Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1064235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1064605Z layer_outputs = layer_module( 2025-12-04T09:45:22.1064935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1065294Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1065672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1066027Z return func(*args, **kwargs) 2025-12-04T09:45:22.1066388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1066762Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1067126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1067482Z return func(*args, **kwargs) 2025-12-04T09:45:22.1067833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1068205Z self_outputs = self.self( 2025-12-04T09:45:22.1068577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1068936Z return func(*args, **kwargs) 2025-12-04T09:45:22.1069303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1069746Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1069925Z 2025-12-04T09:45:22.1070035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1070387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1070694Z res = mod(**inputs) 2025-12-04T09:45:22.1071051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1071433Z outputs = self.roberta( 2025-12-04T09:45:22.1071791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1072176Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1072713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1073098Z layer_outputs = layer_module( 2025-12-04T09:45:22.1073440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1073807Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1074194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1074551Z return func(*args, **kwargs) 2025-12-04T09:45:22.1074922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1075329Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1075693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1076041Z return func(*args, **kwargs) 2025-12-04T09:45:22.1076463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1076886Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1077304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1077677Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1077815Z 2025-12-04T09:45:22.1077916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1078257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1078585Z res = mod(**inputs) 2025-12-04T09:45:22.1078939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1079336Z outputs = self.roberta( 2025-12-04T09:45:22.1079695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1080059Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1080423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1080796Z layer_outputs = layer_module( 2025-12-04T09:45:22.1081117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1081459Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1081818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1082201Z return func(*args, **kwargs) 2025-12-04T09:45:22.1082557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1082956Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1083352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1083737Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1084147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1084614Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1085044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1085435Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1085579Z 2025-12-04T09:45:22.1085679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1086037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1086365Z res = mod(**inputs) 2025-12-04T09:45:22.1086729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1087123Z outputs = self.roberta( 2025-12-04T09:45:22.1087509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1087894Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1088271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1088656Z layer_outputs = layer_module( 2025-12-04T09:45:22.1089001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1089351Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1089762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1090138Z return func(*args, **kwargs) 2025-12-04T09:45:22.1090514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1090911Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1091387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1091824Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1092294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1092855Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1093309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1093737Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1094104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1094441Z return self.act(input) 2025-12-04T09:45:22.1094556Z 2025-12-04T09:45:22.1094660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1095008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1095317Z res = mod(**inputs) 2025-12-04T09:45:22.1095674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1096057Z outputs = self.roberta( 2025-12-04T09:45:22.1096443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1096828Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1097208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1097589Z layer_outputs = layer_module( 2025-12-04T09:45:22.1097923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1098281Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1098652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1099014Z return func(*args, **kwargs) 2025-12-04T09:45:22.1099377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1099770Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1100159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1100534Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1100944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1101413Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1101850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1102233Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1102382Z 2025-12-04T09:45:22.1102480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1102818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1103126Z res = mod(**inputs) 2025-12-04T09:45:22.1103490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1103861Z outputs = self.roberta( 2025-12-04T09:45:22.1104216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1104584Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1104954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1105326Z layer_outputs = layer_module( 2025-12-04T09:45:22.1105657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1105999Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1106387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1106750Z return func(*args, **kwargs) 2025-12-04T09:45:22.1107138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1107523Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1107895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1108249Z return func(*args, **kwargs) 2025-12-04T09:45:22.1108601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1108980Z self_outputs = self.self( 2025-12-04T09:45:22.1109323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1109689Z return func(*args, **kwargs) 2025-12-04T09:45:22.1110048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1110555Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1110806Z 2025-12-04T09:45:22.1110912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1111246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1111551Z res = mod(**inputs) 2025-12-04T09:45:22.1111901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1112268Z outputs = self.roberta( 2025-12-04T09:45:22.1112614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1112990Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1113362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1113728Z layer_outputs = layer_module( 2025-12-04T09:45:22.1114060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1114404Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1114763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1115116Z return func(*args, **kwargs) 2025-12-04T09:45:22.1115477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1115859Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1116220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1116563Z return func(*args, **kwargs) 2025-12-04T09:45:22.1116950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1117324Z self_outputs = self.self( 2025-12-04T09:45:22.1117669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1118034Z return func(*args, **kwargs) 2025-12-04T09:45:22.1118393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1118759Z self.key(current_states) 2025-12-04T09:45:22.1118865Z 2025-12-04T09:45:22.1118963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1119306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1119636Z res = mod(**inputs) 2025-12-04T09:45:22.1120001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1120370Z outputs = self.roberta( 2025-12-04T09:45:22.1120725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1121097Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1121459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1121833Z layer_outputs = layer_module( 2025-12-04T09:45:22.1122162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1122507Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1122858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1123234Z return func(*args, **kwargs) 2025-12-04T09:45:22.1123598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1123977Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1124343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1124696Z return func(*args, **kwargs) 2025-12-04T09:45:22.1125057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1125419Z self_outputs = self.self( 2025-12-04T09:45:22.1125760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1126113Z return func(*args, **kwargs) 2025-12-04T09:45:22.1126477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1126858Z self.value(current_states) 2025-12-04T09:45:22.1126979Z 2025-12-04T09:45:22.1127059Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1127294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1127640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1127962Z res = mod(**inputs) 2025-12-04T09:45:22.1128318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1128696Z outputs = self.roberta( 2025-12-04T09:45:22.1129062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1129450Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1129825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1130209Z layer_outputs = layer_module( 2025-12-04T09:45:22.1130585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1130958Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1131443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1131856Z return func(*args, **kwargs) 2025-12-04T09:45:22.1132276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1132735Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1133143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1133579Z return func(*args, **kwargs) 2025-12-04T09:45:22.1134012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1134409Z self_outputs = self.self( 2025-12-04T09:45:22.1134761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1135160Z return func(*args, **kwargs) 2025-12-04T09:45:22.1135560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1136051Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1136253Z 2025-12-04T09:45:22.1136371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1136733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1137086Z res = mod(**inputs) 2025-12-04T09:45:22.1137449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1137846Z outputs = self.roberta( 2025-12-04T09:45:22.1138226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1138624Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1139008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1139404Z layer_outputs = layer_module( 2025-12-04T09:45:22.1139753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1140110Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1140498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1140879Z return func(*args, **kwargs) 2025-12-04T09:45:22.1141268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1141673Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1142059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1142434Z return func(*args, **kwargs) 2025-12-04T09:45:22.1142809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1143260Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1143710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1144125Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1144263Z 2025-12-04T09:45:22.1144369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1144783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1145100Z res = mod(**inputs) 2025-12-04T09:45:22.1145461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1145841Z outputs = self.roberta( 2025-12-04T09:45:22.1146215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1146605Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1146981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1147372Z layer_outputs = layer_module( 2025-12-04T09:45:22.1147744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1148123Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1148501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1148872Z return func(*args, **kwargs) 2025-12-04T09:45:22.1149252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1149658Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1150051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1150445Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1150873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1151371Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1151817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1152222Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1152363Z 2025-12-04T09:45:22.1152474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1152825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1153146Z res = mod(**inputs) 2025-12-04T09:45:22.1153509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1153898Z outputs = self.roberta( 2025-12-04T09:45:22.1154261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1154657Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1155047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1155433Z layer_outputs = layer_module( 2025-12-04T09:45:22.1155783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1156151Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1156531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1156901Z return func(*args, **kwargs) 2025-12-04T09:45:22.1157279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1157685Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1158075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1158474Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1158927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1159405Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1159822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1160237Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1160610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1160944Z return self.act(input) 2025-12-04T09:45:22.1161051Z 2025-12-04T09:45:22.1161151Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1161522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1161839Z res = mod(**inputs) 2025-12-04T09:45:22.1162209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1162591Z outputs = self.roberta( 2025-12-04T09:45:22.1162955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1163338Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1163708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1164090Z layer_outputs = layer_module( 2025-12-04T09:45:22.1164426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1164771Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1165167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1165535Z return func(*args, **kwargs) 2025-12-04T09:45:22.1165916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1166308Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1166706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1167092Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1167524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1168005Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1168459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1168871Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1169010Z 2025-12-04T09:45:22.1169126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1169480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1169810Z res = mod(**inputs) 2025-12-04T09:45:22.1170178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1170564Z outputs = self.roberta( 2025-12-04T09:45:22.1170959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1171465Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1171891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1172522Z layer_outputs = layer_module( 2025-12-04T09:45:22.1172901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1173355Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1173762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1174159Z return func(*args, **kwargs) 2025-12-04T09:45:22.1174535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1174938Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1175316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1175688Z return func(*args, **kwargs) 2025-12-04T09:45:22.1176101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1176488Z self_outputs = self.self( 2025-12-04T09:45:22.1176924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1177316Z return func(*args, **kwargs) 2025-12-04T09:45:22.1177696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1178225Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1178494Z 2025-12-04T09:45:22.1178600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1178963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1179291Z res = mod(**inputs) 2025-12-04T09:45:22.1179661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1180088Z outputs = self.roberta( 2025-12-04T09:45:22.1180466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1180847Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1181229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1181619Z layer_outputs = layer_module( 2025-12-04T09:45:22.1181963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1182316Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1182692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1183066Z return func(*args, **kwargs) 2025-12-04T09:45:22.1183447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1183841Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1184221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1184655Z return func(*args, **kwargs) 2025-12-04T09:45:22.1185023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1185409Z self_outputs = self.self( 2025-12-04T09:45:22.1185772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1186137Z return func(*args, **kwargs) 2025-12-04T09:45:22.1186505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1186899Z self.key(current_states) 2025-12-04T09:45:22.1187012Z 2025-12-04T09:45:22.1187122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1187496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1187821Z res = mod(**inputs) 2025-12-04T09:45:22.1188186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1188572Z outputs = self.roberta( 2025-12-04T09:45:22.1188938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1189332Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1189715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1190126Z layer_outputs = layer_module( 2025-12-04T09:45:22.1190484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1190853Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1191235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1191602Z return func(*args, **kwargs) 2025-12-04T09:45:22.1191990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1192392Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1192774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1193140Z return func(*args, **kwargs) 2025-12-04T09:45:22.1193521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1193948Z self_outputs = self.self( 2025-12-04T09:45:22.1194307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1194687Z return func(*args, **kwargs) 2025-12-04T09:45:22.1195065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1195455Z self.value(current_states) 2025-12-04T09:45:22.1195569Z 2025-12-04T09:45:22.1195652Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1195893Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1196254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1196567Z res = mod(**inputs) 2025-12-04T09:45:22.1196935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1197327Z outputs = self.roberta( 2025-12-04T09:45:22.1197706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1198092Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1198487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1198872Z layer_outputs = layer_module( 2025-12-04T09:45:22.1199216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1199566Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1199938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1200302Z return func(*args, **kwargs) 2025-12-04T09:45:22.1200680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1201105Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1201511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1201890Z return func(*args, **kwargs) 2025-12-04T09:45:22.1202273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1202727Z self_outputs = self.self( 2025-12-04T09:45:22.1203087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1203449Z return func(*args, **kwargs) 2025-12-04T09:45:22.1203822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1204284Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1204462Z 2025-12-04T09:45:22.1204583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1204925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1205236Z res = mod(**inputs) 2025-12-04T09:45:22.1205591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1205977Z outputs = self.roberta( 2025-12-04T09:45:22.1206339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1206730Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1207113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1207518Z layer_outputs = layer_module( 2025-12-04T09:45:22.1207863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1208225Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1208606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1208971Z return func(*args, **kwargs) 2025-12-04T09:45:22.1209349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1209749Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1210131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1210500Z return func(*args, **kwargs) 2025-12-04T09:45:22.1210879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1211424Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1211913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1212360Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1212521Z 2025-12-04T09:45:22.1212635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1213035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1213370Z res = mod(**inputs) 2025-12-04T09:45:22.1213757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1214173Z outputs = self.roberta( 2025-12-04T09:45:22.1214564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1214992Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1215402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1215848Z layer_outputs = layer_module( 2025-12-04T09:45:22.1216211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1216603Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1217003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1217385Z return func(*args, **kwargs) 2025-12-04T09:45:22.1217787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1218211Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1218664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1219080Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1219561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1220074Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1220546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1220969Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1221122Z 2025-12-04T09:45:22.1221231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1221620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1221958Z res = mod(**inputs) 2025-12-04T09:45:22.1222350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1222776Z outputs = self.roberta( 2025-12-04T09:45:22.1223166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1223571Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1223975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1224387Z layer_outputs = layer_module( 2025-12-04T09:45:22.1224752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1225120Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1225495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1225865Z return func(*args, **kwargs) 2025-12-04T09:45:22.1226234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1226635Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1227032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1227421Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1227835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1228302Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1228736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1229163Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1229542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1229884Z return self.act(input) 2025-12-04T09:45:22.1229996Z 2025-12-04T09:45:22.1230127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1230479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1230800Z res = mod(**inputs) 2025-12-04T09:45:22.1231167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1231552Z outputs = self.roberta( 2025-12-04T09:45:22.1231915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1232302Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1232685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1233091Z layer_outputs = layer_module( 2025-12-04T09:45:22.1233459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1233826Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1234206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1234581Z return func(*args, **kwargs) 2025-12-04T09:45:22.1234957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1235364Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1235760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1236142Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1236584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1237071Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1237515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1237910Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1238049Z 2025-12-04T09:45:22.1238151Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1238500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1238810Z res = mod(**inputs) 2025-12-04T09:45:22.1239171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1239561Z outputs = self.roberta( 2025-12-04T09:45:22.1239939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1240328Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1240706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1241084Z layer_outputs = layer_module( 2025-12-04T09:45:22.1241414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1241770Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1242142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1242512Z return func(*args, **kwargs) 2025-12-04T09:45:22.1242886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1243301Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1243673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1244051Z return func(*args, **kwargs) 2025-12-04T09:45:22.1244433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1244819Z self_outputs = self.self( 2025-12-04T09:45:22.1245178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1245538Z return func(*args, **kwargs) 2025-12-04T09:45:22.1245915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1246444Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1246727Z 2025-12-04T09:45:22.1246838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1247219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1247544Z res = mod(**inputs) 2025-12-04T09:45:22.1247915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1248318Z outputs = self.roberta( 2025-12-04T09:45:22.1248723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1249144Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1249555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1249964Z layer_outputs = layer_module( 2025-12-04T09:45:22.1250337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1250744Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1251273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1251716Z return func(*args, **kwargs) 2025-12-04T09:45:22.1252135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1252583Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1252961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1253339Z return func(*args, **kwargs) 2025-12-04T09:45:22.1253711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1254095Z self_outputs = self.self( 2025-12-04T09:45:22.1254445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1254815Z return func(*args, **kwargs) 2025-12-04T09:45:22.1255200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1255573Z self.key(current_states) 2025-12-04T09:45:22.1255693Z 2025-12-04T09:45:22.1255794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1256144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1256451Z res = mod(**inputs) 2025-12-04T09:45:22.1256798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1257178Z outputs = self.roberta( 2025-12-04T09:45:22.1257540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1257917Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1258317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1258700Z layer_outputs = layer_module( 2025-12-04T09:45:22.1259039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1259386Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1259751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1260106Z return func(*args, **kwargs) 2025-12-04T09:45:22.1260469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1260879Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1261247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1261622Z return func(*args, **kwargs) 2025-12-04T09:45:22.1261987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1262366Z self_outputs = self.self( 2025-12-04T09:45:22.1262718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1263078Z return func(*args, **kwargs) 2025-12-04T09:45:22.1263438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1263817Z self.value(current_states) 2025-12-04T09:45:22.1263930Z 2025-12-04T09:45:22.1264016Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1264271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1264625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1264942Z res = mod(**inputs) 2025-12-04T09:45:22.1265301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1265673Z outputs = self.roberta( 2025-12-04T09:45:22.1266034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1266414Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1266782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1267166Z layer_outputs = layer_module( 2025-12-04T09:45:22.1267504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1267863Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1268226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1268596Z return func(*args, **kwargs) 2025-12-04T09:45:22.1268953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1269332Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1269561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1269624Z return func(*args, **kwargs) 2025-12-04T09:45:22.1269875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1269940Z self_outputs = self.self( 2025-12-04T09:45:22.1270169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1270241Z return func(*args, **kwargs) 2025-12-04T09:45:22.1270511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1270647Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1270651Z 2025-12-04T09:45:22.1270751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1270939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1271008Z res = mod(**inputs) 2025-12-04T09:45:22.1271254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1271325Z outputs = self.roberta( 2025-12-04T09:45:22.1271573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1271662Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1271939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1272010Z layer_outputs = layer_module( 2025-12-04T09:45:22.1272231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1272438Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1272676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1272750Z return func(*args, **kwargs) 2025-12-04T09:45:22.1272993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1273069Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1273372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1273437Z return func(*args, **kwargs) 2025-12-04T09:45:22.1273682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1273812Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1274054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1274138Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1274142Z 2025-12-04T09:45:22.1274239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1274424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1274493Z res = mod(**inputs) 2025-12-04T09:45:22.1274739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1274814Z outputs = self.roberta( 2025-12-04T09:45:22.1275056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1275125Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1275376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1275441Z layer_outputs = layer_module( 2025-12-04T09:45:22.1275647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1275726Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1275953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1276028Z return func(*args, **kwargs) 2025-12-04T09:45:22.1276272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1276378Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1276631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1276704Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1276992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1277106Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1277351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1277435Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1277477Z 2025-12-04T09:45:22.1277577Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1277796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1277860Z res = mod(**inputs) 2025-12-04T09:45:22.1278112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1278183Z outputs = self.roberta( 2025-12-04T09:45:22.1278439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1278506Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1278765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1278832Z layer_outputs = layer_module( 2025-12-04T09:45:22.1279053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1279144Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1279377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1279450Z return func(*args, **kwargs) 2025-12-04T09:45:22.1279696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1279772Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1280027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1280098Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1280385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1280501Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1280747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1280861Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1281065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1281138Z return self.act(input) 2025-12-04T09:45:22.1281141Z 2025-12-04T09:45:22.1281238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1281425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1281491Z res = mod(**inputs) 2025-12-04T09:45:22.1281734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1281800Z outputs = self.roberta( 2025-12-04T09:45:22.1282050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1282120Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1282389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1282459Z layer_outputs = layer_module( 2025-12-04T09:45:22.1282669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1282750Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1282979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1283043Z return func(*args, **kwargs) 2025-12-04T09:45:22.1283295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1283402Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1283664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1283737Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1284012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1284142Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1284388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1284469Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1284473Z 2025-12-04T09:45:22.1284570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1284753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1284838Z res = mod(**inputs) 2025-12-04T09:45:22.1285086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1285156Z outputs = self.roberta( 2025-12-04T09:45:22.1285399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1285466Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1285714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1285780Z layer_outputs = layer_module( 2025-12-04T09:45:22.1285989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1286069Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1286300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1286372Z return func(*args, **kwargs) 2025-12-04T09:45:22.1286622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1286699Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1286933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1286997Z return func(*args, **kwargs) 2025-12-04T09:45:22.1287242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1287316Z self_outputs = self.self( 2025-12-04T09:45:22.1287551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1287623Z return func(*args, **kwargs) 2025-12-04T09:45:22.1287869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1288129Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1288133Z 2025-12-04T09:45:22.1288242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1288428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1288496Z res = mod(**inputs) 2025-12-04T09:45:22.1288743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1288806Z outputs = self.roberta( 2025-12-04T09:45:22.1289060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1289150Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1289410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1289487Z layer_outputs = layer_module( 2025-12-04T09:45:22.1289695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1289774Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1289998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1290063Z return func(*args, **kwargs) 2025-12-04T09:45:22.1290316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1290393Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1290632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1290721Z return func(*args, **kwargs) 2025-12-04T09:45:22.1290980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1291061Z self_outputs = self.self( 2025-12-04T09:45:22.1291389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1291465Z return func(*args, **kwargs) 2025-12-04T09:45:22.1291752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1291823Z self.key(current_states) 2025-12-04T09:45:22.1291827Z 2025-12-04T09:45:22.1291942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1292148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1292217Z res = mod(**inputs) 2025-12-04T09:45:22.1292533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1292601Z outputs = self.roberta( 2025-12-04T09:45:22.1292862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1292940Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1293208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1293289Z layer_outputs = layer_module( 2025-12-04T09:45:22.1293524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1293603Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1293868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1293943Z return func(*args, **kwargs) 2025-12-04T09:45:22.1294256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1294345Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1294602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1294680Z return func(*args, **kwargs) 2025-12-04T09:45:22.1294958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1295031Z self_outputs = self.self( 2025-12-04T09:45:22.1295292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1295363Z return func(*args, **kwargs) 2025-12-04T09:45:22.1295673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1295770Z self.value(current_states) 2025-12-04T09:45:22.1295774Z 2025-12-04T09:45:22.1295864Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1295981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1296189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1296257Z res = mod(**inputs) 2025-12-04T09:45:22.1296536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1296607Z outputs = self.roberta( 2025-12-04T09:45:22.1296886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1296961Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1297262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1297347Z layer_outputs = layer_module( 2025-12-04T09:45:22.1297582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1297671Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1297922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1297991Z return func(*args, **kwargs) 2025-12-04T09:45:22.1298272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1298355Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1298606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1298686Z return func(*args, **kwargs) 2025-12-04T09:45:22.1298960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1299038Z self_outputs = self.self( 2025-12-04T09:45:22.1299292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1299368Z return func(*args, **kwargs) 2025-12-04T09:45:22.1299626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1299756Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1299759Z 2025-12-04T09:45:22.1299865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1300059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1300124Z res = mod(**inputs) 2025-12-04T09:45:22.1300385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1300449Z outputs = self.roberta( 2025-12-04T09:45:22.1300715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1300795Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1301048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1301124Z layer_outputs = layer_module( 2025-12-04T09:45:22.1301338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1301413Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1301653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1301735Z return func(*args, **kwargs) 2025-12-04T09:45:22.1302011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1302099Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1302334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1302405Z return func(*args, **kwargs) 2025-12-04T09:45:22.1302656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1302778Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1303034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1303112Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1303132Z 2025-12-04T09:45:22.1303240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1303434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1303497Z res = mod(**inputs) 2025-12-04T09:45:22.1303756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1303821Z outputs = self.roberta( 2025-12-04T09:45:22.1304072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1304147Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1304397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1304473Z layer_outputs = layer_module( 2025-12-04T09:45:22.1304690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1304763Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1305013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1305079Z return func(*args, **kwargs) 2025-12-04T09:45:22.1305330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1305422Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1305669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1305749Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1306034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1306151Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1306411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1306517Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1306521Z 2025-12-04T09:45:22.1306628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1306819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1306879Z res = mod(**inputs) 2025-12-04T09:45:22.1307139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1307202Z outputs = self.roberta( 2025-12-04T09:45:22.1307458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1307547Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1307818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1307898Z layer_outputs = layer_module( 2025-12-04T09:45:22.1308116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1308189Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1308435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1308500Z return func(*args, **kwargs) 2025-12-04T09:45:22.1308760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1308840Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1309089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1309193Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1309480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1309594Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1309854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1309962Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1310175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1310242Z return self.act(input) 2025-12-04T09:45:22.1310245Z 2025-12-04T09:45:22.1310343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1310537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1310599Z res = mod(**inputs) 2025-12-04T09:45:22.1310860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1310924Z outputs = self.roberta( 2025-12-04T09:45:22.1311176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1311252Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1311501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1311570Z layer_outputs = layer_module( 2025-12-04T09:45:22.1311793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1311866Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1312113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1312185Z return func(*args, **kwargs) 2025-12-04T09:45:22.1312444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1312533Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1312777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1312853Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1313130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1313255Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1313530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1313624Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1313627Z 2025-12-04T09:45:22.1313738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1313938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1313998Z res = mod(**inputs) 2025-12-04T09:45:22.1314250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1314313Z outputs = self.roberta( 2025-12-04T09:45:22.1314559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1314636Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1314882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1314973Z layer_outputs = layer_module( 2025-12-04T09:45:22.1315186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1315260Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1315498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1315563Z return func(*args, **kwargs) 2025-12-04T09:45:22.1315809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1315892Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1316122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1316193Z return func(*args, **kwargs) 2025-12-04T09:45:22.1316442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1316509Z self_outputs = self.self( 2025-12-04T09:45:22.1316748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1316811Z return func(*args, **kwargs) 2025-12-04T09:45:22.1317056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1317259Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1317262Z 2025-12-04T09:45:22.1317360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1317552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1317611Z res = mod(**inputs) 2025-12-04T09:45:22.1317859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1317930Z outputs = self.roberta( 2025-12-04T09:45:22.1318192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1318265Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1318516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1318583Z layer_outputs = layer_module( 2025-12-04T09:45:22.1318806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1318880Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1319118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1319208Z return func(*args, **kwargs) 2025-12-04T09:45:22.1319464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1319565Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1319802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1319868Z return func(*args, **kwargs) 2025-12-04T09:45:22.1320126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1320193Z self_outputs = self.self( 2025-12-04T09:45:22.1320432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1320496Z return func(*args, **kwargs) 2025-12-04T09:45:22.1320746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1320838Z self.key(current_states) 2025-12-04T09:45:22.1320842Z 2025-12-04T09:45:22.1320943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1321136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1321202Z res = mod(**inputs) 2025-12-04T09:45:22.1321453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1321522Z outputs = self.roberta( 2025-12-04T09:45:22.1321772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1321839Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1322096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1322166Z layer_outputs = layer_module( 2025-12-04T09:45:22.1322382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1322465Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1322703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1322775Z return func(*args, **kwargs) 2025-12-04T09:45:22.1323029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1323106Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1323360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1323423Z return func(*args, **kwargs) 2025-12-04T09:45:22.1323675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1323743Z self_outputs = self.self( 2025-12-04T09:45:22.1323971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1324057Z return func(*args, **kwargs) 2025-12-04T09:45:22.1324306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1324374Z self.value(current_states) 2025-12-04T09:45:22.1324385Z 2025-12-04T09:45:22.1324464Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1324561Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1324753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1324812Z res = mod(**inputs) 2025-12-04T09:45:22.1325059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1325153Z outputs = self.roberta( 2025-12-04T09:45:22.1325419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1325490Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1325750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1325817Z layer_outputs = layer_module( 2025-12-04T09:45:22.1326039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1326113Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1326354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1326427Z return func(*args, **kwargs) 2025-12-04T09:45:22.1326707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1326794Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1327029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1327095Z return func(*args, **kwargs) 2025-12-04T09:45:22.1327354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1327418Z self_outputs = self.self( 2025-12-04T09:45:22.1327650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1327723Z return func(*args, **kwargs) 2025-12-04T09:45:22.1327971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1328108Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1328111Z 2025-12-04T09:45:22.1328209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1328399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1328466Z res = mod(**inputs) 2025-12-04T09:45:22.1328714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1328786Z outputs = self.roberta( 2025-12-04T09:45:22.1329034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1329101Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1329357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1329427Z layer_outputs = layer_module( 2025-12-04T09:45:22.1329640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1329723Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1329981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1330056Z return func(*args, **kwargs) 2025-12-04T09:45:22.1330306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1330382Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1330623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1330686Z return func(*args, **kwargs) 2025-12-04T09:45:22.1330938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1331115Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1332282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1332389Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1332394Z 2025-12-04T09:45:22.1332499Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1332695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1332767Z res = mod(**inputs) 2025-12-04T09:45:22.1333036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1333112Z outputs = self.roberta( 2025-12-04T09:45:22.1333366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1333462Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1333722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1333793Z layer_outputs = layer_module( 2025-12-04T09:45:22.1334008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1334088Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1334322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1334393Z return func(*args, **kwargs) 2025-12-04T09:45:22.1334643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1334720Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1334977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1335051Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1335343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1335458Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1335705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1335790Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1335793Z 2025-12-04T09:45:22.1335890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1336079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1336146Z res = mod(**inputs) 2025-12-04T09:45:22.1336397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1336470Z outputs = self.roberta( 2025-12-04T09:45:22.1336742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1336813Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1337069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1337138Z layer_outputs = layer_module( 2025-12-04T09:45:22.1337356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1337430Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1337660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1337757Z return func(*args, **kwargs) 2025-12-04T09:45:22.1338010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1338109Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1338370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1338444Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1338735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1338848Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1339098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1339214Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1339440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1339515Z return self.act(input) 2025-12-04T09:45:22.1339520Z 2025-12-04T09:45:22.1339622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1339811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1339878Z res = mod(**inputs) 2025-12-04T09:45:22.1340133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1340199Z outputs = self.roberta( 2025-12-04T09:45:22.1340457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1340526Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1340783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1340855Z layer_outputs = layer_module( 2025-12-04T09:45:22.1341073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1341157Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1341393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1341459Z return func(*args, **kwargs) 2025-12-04T09:45:22.1341718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1341798Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1342054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1342126Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1342414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1342552Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1342824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1342911Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1342915Z 2025-12-04T09:45:22.1343015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1343202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1343270Z res = mod(**inputs) 2025-12-04T09:45:22.1343521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1343586Z outputs = self.roberta( 2025-12-04T09:45:22.1343873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1343959Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1344217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1344285Z layer_outputs = layer_module( 2025-12-04T09:45:22.1344499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1344581Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1344813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1344884Z return func(*args, **kwargs) 2025-12-04T09:45:22.1345131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1345229Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1345471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1345538Z return func(*args, **kwargs) 2025-12-04T09:45:22.1345789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1345864Z self_outputs = self.self( 2025-12-04T09:45:22.1346096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1346167Z return func(*args, **kwargs) 2025-12-04T09:45:22.1346418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1346622Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1346628Z 2025-12-04T09:45:22.1346735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1346926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1346998Z res = mod(**inputs) 2025-12-04T09:45:22.1347249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1347313Z outputs = self.roberta( 2025-12-04T09:45:22.1347569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1347639Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1347889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1347965Z layer_outputs = layer_module( 2025-12-04T09:45:22.1348179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1348264Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1348520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1348586Z return func(*args, **kwargs) 2025-12-04T09:45:22.1348849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1348929Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1349176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1349241Z return func(*args, **kwargs) 2025-12-04T09:45:22.1349494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1349588Z self_outputs = self.self( 2025-12-04T09:45:22.1349832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1349926Z return func(*args, **kwargs) 2025-12-04T09:45:22.1350185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1350251Z self.key(current_states) 2025-12-04T09:45:22.1350254Z 2025-12-04T09:45:22.1350360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1350549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1350609Z res = mod(**inputs) 2025-12-04T09:45:22.1350860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1350922Z outputs = self.roberta( 2025-12-04T09:45:22.1351171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1351268Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1351521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1351594Z layer_outputs = layer_module( 2025-12-04T09:45:22.1351810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1351881Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1352120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1352184Z return func(*args, **kwargs) 2025-12-04T09:45:22.1352441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1352518Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1352751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1352823Z return func(*args, **kwargs) 2025-12-04T09:45:22.1353073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1353138Z self_outputs = self.self( 2025-12-04T09:45:22.1353387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1353450Z return func(*args, **kwargs) 2025-12-04T09:45:22.1353702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1353768Z self.value(current_states) 2025-12-04T09:45:22.1353771Z 2025-12-04T09:45:22.1353849Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1353957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1354147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1354211Z res = mod(**inputs) 2025-12-04T09:45:22.1354494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1357322Z outputs = self.roberta( 2025-12-04T09:45:22.1357597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1357675Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1357949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1358024Z layer_outputs = layer_module( 2025-12-04T09:45:22.1358242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1358319Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1358586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1358657Z return func(*args, **kwargs) 2025-12-04T09:45:22.1358907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1359021Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1359254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1359319Z return func(*args, **kwargs) 2025-12-04T09:45:22.1359566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1359638Z self_outputs = self.self( 2025-12-04T09:45:22.1359868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1359955Z return func(*args, **kwargs) 2025-12-04T09:45:22.1360217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1360344Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1360349Z 2025-12-04T09:45:22.1360458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1360649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1360708Z res = mod(**inputs) 2025-12-04T09:45:22.1360970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1361036Z outputs = self.roberta( 2025-12-04T09:45:22.1361291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1361362Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1361617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1361690Z layer_outputs = layer_module( 2025-12-04T09:45:22.1361905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1361978Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1362228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1362293Z return func(*args, **kwargs) 2025-12-04T09:45:22.1362559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1362636Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1362879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1362953Z return func(*args, **kwargs) 2025-12-04T09:45:22.1363233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1363367Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1363691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1363769Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1363773Z 2025-12-04T09:45:22.1363888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1364074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1364135Z res = mod(**inputs) 2025-12-04T09:45:22.1364387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1364449Z outputs = self.roberta( 2025-12-04T09:45:22.1364718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1364788Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1365035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1365111Z layer_outputs = layer_module( 2025-12-04T09:45:22.1365323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1365403Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1365637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1365723Z return func(*args, **kwargs) 2025-12-04T09:45:22.1365991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1366077Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1366330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1366413Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1366705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1366827Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1367086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1367164Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1367169Z 2025-12-04T09:45:22.1367274Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1367468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1367537Z res = mod(**inputs) 2025-12-04T09:45:22.1367795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1367861Z outputs = self.roberta( 2025-12-04T09:45:22.1368123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1368191Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1368447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1368522Z layer_outputs = layer_module( 2025-12-04T09:45:22.1368738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1368820Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1369080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1369147Z return func(*args, **kwargs) 2025-12-04T09:45:22.1369404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1369521Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1369767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1369848Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1370134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1370255Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1370518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1370630Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1370848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1370921Z return self.act(input) 2025-12-04T09:45:22.1370925Z 2025-12-04T09:45:22.1371043Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1371363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1371438Z res = mod(**inputs) 2025-12-04T09:45:22.1371733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1371809Z outputs = self.roberta( 2025-12-04T09:45:22.1372130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1372215Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1372776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1372864Z layer_outputs = layer_module( 2025-12-04T09:45:22.1373103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1373183Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1373459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1373525Z return func(*args, **kwargs) 2025-12-04T09:45:22.1373786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1373869Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1374120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1374204Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1374491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1374624Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1374884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1374962Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1374966Z 2025-12-04T09:45:22.1375074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1375266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1375330Z res = mod(**inputs) 2025-12-04T09:45:22.1375594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1375704Z outputs = self.roberta( 2025-12-04T09:45:22.1375966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1376080Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1376355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1376423Z layer_outputs = layer_module( 2025-12-04T09:45:22.1376645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1376721Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1376961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1377035Z return func(*args, **kwargs) 2025-12-04T09:45:22.1377319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1377400Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1377642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1377707Z return func(*args, **kwargs) 2025-12-04T09:45:22.1377967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1378033Z self_outputs = self.self( 2025-12-04T09:45:22.1378267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1378372Z return func(*args, **kwargs) 2025-12-04T09:45:22.1378633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1378850Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1378854Z 2025-12-04T09:45:22.1378957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1379157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1379224Z res = mod(**inputs) 2025-12-04T09:45:22.1379484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1379548Z outputs = self.roberta( 2025-12-04T09:45:22.1379813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1379883Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1380150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1380219Z layer_outputs = layer_module( 2025-12-04T09:45:22.1380441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1380526Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1380766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1380840Z return func(*args, **kwargs) 2025-12-04T09:45:22.1381095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1381175Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1381422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1381490Z return func(*args, **kwargs) 2025-12-04T09:45:22.1381767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1381841Z self_outputs = self.self( 2025-12-04T09:45:22.1382076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1382171Z return func(*args, **kwargs) 2025-12-04T09:45:22.1382424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1382490Z self.key(current_states) 2025-12-04T09:45:22.1382494Z 2025-12-04T09:45:22.1382602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1382792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1382854Z res = mod(**inputs) 2025-12-04T09:45:22.1383139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1383203Z outputs = self.roberta( 2025-12-04T09:45:22.1383453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1383521Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1383762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1383834Z layer_outputs = layer_module( 2025-12-04T09:45:22.1384039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1384117Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1384343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1384427Z return func(*args, **kwargs) 2025-12-04T09:45:22.1384682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1384760Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1384986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1385057Z return func(*args, **kwargs) 2025-12-04T09:45:22.1385301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1385373Z self_outputs = self.self( 2025-12-04T09:45:22.1385600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1385661Z return func(*args, **kwargs) 2025-12-04T09:45:22.1385913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1385980Z self.value(current_states) 2025-12-04T09:45:22.1385983Z 2025-12-04T09:45:22.1386070Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1386167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1386349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1386419Z res = mod(**inputs) 2025-12-04T09:45:22.1386662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1386724Z outputs = self.roberta( 2025-12-04T09:45:22.1386980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1387049Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1387310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1387379Z layer_outputs = layer_module( 2025-12-04T09:45:22.1387624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1387707Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1387959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1388022Z return func(*args, **kwargs) 2025-12-04T09:45:22.1388275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1388351Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1388588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1388652Z return func(*args, **kwargs) 2025-12-04T09:45:22.1388917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1388994Z self_outputs = self.self( 2025-12-04T09:45:22.1389224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1389290Z return func(*args, **kwargs) 2025-12-04T09:45:22.1389543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1389671Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1389674Z 2025-12-04T09:45:22.1389777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1389959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1390040Z res = mod(**inputs) 2025-12-04T09:45:22.1390292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1390356Z outputs = self.roberta( 2025-12-04T09:45:22.1390607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1390675Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1390916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1390988Z layer_outputs = layer_module( 2025-12-04T09:45:22.1391195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1391265Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1391500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1391564Z return func(*args, **kwargs) 2025-12-04T09:45:22.1391818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1391893Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1392117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1392190Z return func(*args, **kwargs) 2025-12-04T09:45:22.1392431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1392560Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1392803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1392879Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1392884Z 2025-12-04T09:45:22.1392986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1393172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1393267Z res = mod(**inputs) 2025-12-04T09:45:22.1393525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1393614Z outputs = self.roberta( 2025-12-04T09:45:22.1393880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1393950Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1394214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1394290Z layer_outputs = layer_module( 2025-12-04T09:45:22.1394519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1394602Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1394862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1394928Z return func(*args, **kwargs) 2025-12-04T09:45:22.1395196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1395275Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1395527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1395606Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1395900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1396052Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1396309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1396392Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1396396Z 2025-12-04T09:45:22.1396501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1396692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1396761Z res = mod(**inputs) 2025-12-04T09:45:22.1397013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1397077Z outputs = self.roberta( 2025-12-04T09:45:22.1397334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1397407Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1397666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1397743Z layer_outputs = layer_module( 2025-12-04T09:45:22.1397964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1398049Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1398291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1398357Z return func(*args, **kwargs) 2025-12-04T09:45:22.1398622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1398714Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1398962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1399042Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1399370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1399492Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1399764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1399872Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1400087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1400153Z return self.act(input) 2025-12-04T09:45:22.1400156Z 2025-12-04T09:45:22.1400261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1400448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1400511Z res = mod(**inputs) 2025-12-04T09:45:22.1400787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1400853Z outputs = self.roberta( 2025-12-04T09:45:22.1401108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1401184Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1401434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1401508Z layer_outputs = layer_module( 2025-12-04T09:45:22.1401722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1401794Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1402063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1402129Z return func(*args, **kwargs) 2025-12-04T09:45:22.1402391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1402470Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1402720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1402801Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1403086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1403213Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1403475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1403557Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1403560Z 2025-12-04T09:45:22.1403671Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1403867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1403930Z res = mod(**inputs) 2025-12-04T09:45:22.1404198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1404264Z outputs = self.roberta( 2025-12-04T09:45:22.1404530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1404609Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1404977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1405054Z layer_outputs = layer_module( 2025-12-04T09:45:22.1405269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1405361Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1405603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1405692Z return func(*args, **kwargs) 2025-12-04T09:45:22.1405955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1406034Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1406269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1406347Z return func(*args, **kwargs) 2025-12-04T09:45:22.1406607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1406686Z self_outputs = self.self( 2025-12-04T09:45:22.1406945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1407015Z return func(*args, **kwargs) 2025-12-04T09:45:22.1407279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1407487Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1407490Z 2025-12-04T09:45:22.1407593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1407794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1407856Z res = mod(**inputs) 2025-12-04T09:45:22.1408125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1408213Z outputs = self.roberta( 2025-12-04T09:45:22.1408481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1408562Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1408825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1408903Z layer_outputs = layer_module( 2025-12-04T09:45:22.1409126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1409202Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1409466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1409536Z return func(*args, **kwargs) 2025-12-04T09:45:22.1409815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1409910Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1410169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1410250Z return func(*args, **kwargs) 2025-12-04T09:45:22.1410529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1410603Z self_outputs = self.self( 2025-12-04T09:45:22.1410869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1410942Z return func(*args, **kwargs) 2025-12-04T09:45:22.1411326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1411418Z self.key(current_states) 2025-12-04T09:45:22.1411422Z 2025-12-04T09:45:22.1411537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1411789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1411860Z res = mod(**inputs) 2025-12-04T09:45:22.1412144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1412255Z outputs = self.roberta( 2025-12-04T09:45:22.1412534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1412614Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1412874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1412944Z layer_outputs = layer_module( 2025-12-04T09:45:22.1413176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1413270Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1413513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1413599Z return func(*args, **kwargs) 2025-12-04T09:45:22.1413851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1413935Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1414171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1414236Z return func(*args, **kwargs) 2025-12-04T09:45:22.1414491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1414576Z self_outputs = self.self( 2025-12-04T09:45:22.1414810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1414883Z return func(*args, **kwargs) 2025-12-04T09:45:22.1415131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1415209Z self.value(current_states) 2025-12-04T09:45:22.1415212Z 2025-12-04T09:45:22.1415290Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1415387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1415583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1415644Z res = mod(**inputs) 2025-12-04T09:45:22.1415900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1415967Z outputs = self.roberta( 2025-12-04T09:45:22.1416217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1416294Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1416543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1416614Z layer_outputs = layer_module( 2025-12-04T09:45:22.1416834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1416907Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1417146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1417211Z return func(*args, **kwargs) 2025-12-04T09:45:22.1417460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1417547Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1417802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1417870Z return func(*args, **kwargs) 2025-12-04T09:45:22.1418153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1418221Z self_outputs = self.self( 2025-12-04T09:45:22.1418459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1418523Z return func(*args, **kwargs) 2025-12-04T09:45:22.1418772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1418911Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1418915Z 2025-12-04T09:45:22.1419015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1419236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1419300Z res = mod(**inputs) 2025-12-04T09:45:22.1419570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1419643Z outputs = self.roberta( 2025-12-04T09:45:22.1419895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1419961Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1420217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1420283Z layer_outputs = layer_module( 2025-12-04T09:45:22.1420530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1420606Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1420848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1420920Z return func(*args, **kwargs) 2025-12-04T09:45:22.1421179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1421264Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1421503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1421567Z return func(*args, **kwargs) 2025-12-04T09:45:22.1421832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1421962Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1422221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1422307Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1422311Z 2025-12-04T09:45:22.1422411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1422616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1422679Z res = mod(**inputs) 2025-12-04T09:45:22.1422936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1423010Z outputs = self.roberta( 2025-12-04T09:45:22.1423275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1423352Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1423605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1423695Z layer_outputs = layer_module( 2025-12-04T09:45:22.1423916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1424008Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1424240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1424316Z return func(*args, **kwargs) 2025-12-04T09:45:22.1424574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1424665Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1424920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1424997Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1425315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1425435Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1425713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1425796Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1425799Z 2025-12-04T09:45:22.1425902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1426107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1426168Z res = mod(**inputs) 2025-12-04T09:45:22.1426428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1426521Z outputs = self.roberta( 2025-12-04T09:45:22.1426781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1426858Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1427114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1427184Z layer_outputs = layer_module( 2025-12-04T09:45:22.1427412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1427485Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1427728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1427799Z return func(*args, **kwargs) 2025-12-04T09:45:22.1428053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1428142Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1428396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1428471Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1428769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1428884Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1429154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1429263Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1429473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1429550Z return self.act(input) 2025-12-04T09:45:22.1429553Z 2025-12-04T09:45:22.1429671Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1429869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1429959Z res = mod(**inputs) 2025-12-04T09:45:22.1430217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1430290Z outputs = self.roberta( 2025-12-04T09:45:22.1430555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1430625Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1430896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1430968Z layer_outputs = layer_module( 2025-12-04T09:45:22.1431220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1431301Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1431543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1431619Z return func(*args, **kwargs) 2025-12-04T09:45:22.1431879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1431960Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1432224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1432297Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1432613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1432747Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1433010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1433103Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1433106Z 2025-12-04T09:45:22.1433208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1433413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1433477Z res = mod(**inputs) 2025-12-04T09:45:22.1433734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1433808Z outputs = self.roberta( 2025-12-04T09:45:22.1434065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1434138Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1434404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1434473Z layer_outputs = layer_module( 2025-12-04T09:45:22.1434702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1434776Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1435017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1435093Z return func(*args, **kwargs) 2025-12-04T09:45:22.1435347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1435430Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1435681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1435749Z return func(*args, **kwargs) 2025-12-04T09:45:22.1436073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1436157Z self_outputs = self.self( 2025-12-04T09:45:22.1436400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1436473Z return func(*args, **kwargs) 2025-12-04T09:45:22.1436729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T09:45:22.1436946Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T09:45:22.1436951Z 2025-12-04T09:45:22.1437054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1437281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1437356Z res = mod(**inputs) 2025-12-04T09:45:22.1437618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1437687Z outputs = self.roberta( 2025-12-04T09:45:22.1437952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1438024Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1438288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1438357Z layer_outputs = layer_module( 2025-12-04T09:45:22.1438576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1438683Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1438927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1439001Z return func(*args, **kwargs) 2025-12-04T09:45:22.1439259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1439340Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1439590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1439657Z return func(*args, **kwargs) 2025-12-04T09:45:22.1439918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1439996Z self_outputs = self.self( 2025-12-04T09:45:22.1440238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1440314Z return func(*args, **kwargs) 2025-12-04T09:45:22.1440574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T09:45:22.1440644Z self.key(current_states) 2025-12-04T09:45:22.1440649Z 2025-12-04T09:45:22.1440759Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1440953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1441022Z res = mod(**inputs) 2025-12-04T09:45:22.1441280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1441346Z outputs = self.roberta( 2025-12-04T09:45:22.1441609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1441683Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1441964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1442044Z layer_outputs = layer_module( 2025-12-04T09:45:22.1442275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1442375Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1442609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1442672Z return func(*args, **kwargs) 2025-12-04T09:45:22.1442928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1443006Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1443236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1443332Z return func(*args, **kwargs) 2025-12-04T09:45:22.1443589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1443661Z self_outputs = self.self( 2025-12-04T09:45:22.1443894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1443957Z return func(*args, **kwargs) 2025-12-04T09:45:22.1444216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T09:45:22.1444284Z self.value(current_states) 2025-12-04T09:45:22.1444287Z 2025-12-04T09:45:22.1444372Z cudagraph partition due to non gpu ops 2025-12-04T09:45:22.1444472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1444686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1444755Z res = mod(**inputs) 2025-12-04T09:45:22.1445008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1445074Z outputs = self.roberta( 2025-12-04T09:45:22.1445338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1445405Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1445663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1445730Z layer_outputs = layer_module( 2025-12-04T09:45:22.1445944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1446026Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1446260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1446327Z return func(*args, **kwargs) 2025-12-04T09:45:22.1446587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1446664Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1446906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1446971Z return func(*args, **kwargs) 2025-12-04T09:45:22.1447220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T09:45:22.1447295Z self_outputs = self.self( 2025-12-04T09:45:22.1447529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1447605Z return func(*args, **kwargs) 2025-12-04T09:45:22.1447884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T09:45:22.1448022Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T09:45:22.1448042Z 2025-12-04T09:45:22.1448154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1448347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1448411Z res = mod(**inputs) 2025-12-04T09:45:22.1448675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1448741Z outputs = self.roberta( 2025-12-04T09:45:22.1449007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1449080Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1449359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1449437Z layer_outputs = layer_module( 2025-12-04T09:45:22.1449661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1449739Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1449991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1450057Z return func(*args, **kwargs) 2025-12-04T09:45:22.1450322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T09:45:22.1450402Z self_attention_outputs = self.attention( 2025-12-04T09:45:22.1450659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1450735Z return func(*args, **kwargs) 2025-12-04T09:45:22.1450995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T09:45:22.1451128Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:45:22.1451642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T09:45:22.1451734Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1451739Z 2025-12-04T09:45:22.1451858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1452064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1452130Z res = mod(**inputs) 2025-12-04T09:45:22.1452414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1452490Z outputs = self.roberta( 2025-12-04T09:45:22.1452773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1452859Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1453117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1453196Z layer_outputs = layer_module( 2025-12-04T09:45:22.1453420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1453504Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1453750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1453820Z return func(*args, **kwargs) 2025-12-04T09:45:22.1454083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1454193Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1454453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1454558Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1454855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1454981Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1455244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T09:45:22.1455325Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1455331Z 2025-12-04T09:45:22.1455442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1455660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1455731Z res = mod(**inputs) 2025-12-04T09:45:22.1455996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1456066Z outputs = self.roberta( 2025-12-04T09:45:22.1456329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1456398Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1456654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1456730Z layer_outputs = layer_module( 2025-12-04T09:45:22.1456950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1457050Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1457296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1457363Z return func(*args, **kwargs) 2025-12-04T09:45:22.1457630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1457712Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1457965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1458046Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1458336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T09:45:22.1458460Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:45:22.1458718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T09:45:22.1458830Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:45:22.1459051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:45:22.1459119Z return self.act(input) 2025-12-04T09:45:22.1459122Z 2025-12-04T09:45:22.1459229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1459421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1459483Z res = mod(**inputs) 2025-12-04T09:45:22.1459746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T09:45:22.1459812Z outputs = self.roberta( 2025-12-04T09:45:22.1460069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T09:45:22.1460148Z encoder_outputs = self.encoder( 2025-12-04T09:45:22.1460419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T09:45:22.1460497Z layer_outputs = layer_module( 2025-12-04T09:45:22.1460761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:22.1460836Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:22.1461086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:22.1461154Z return func(*args, **kwargs) 2025-12-04T09:45:22.1461416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T09:45:22.1461499Z layer_output = apply_chunking_to_forward( 2025-12-04T09:45:22.1461776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:45:22.1461861Z return forward_fn(*input_tensors) 2025-12-04T09:45:22.1462158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T09:45:22.1462301Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:45:22.1462554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T09:45:22.1462632Z hidden_states = self.dense(hidden_states) 2025-12-04T09:45:22.1462635Z 2025-12-04T09:45:22.1462739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1462927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1463007Z res = mod(**inputs) 2025-12-04T09:45:22.1463264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 988, in forward 2025-12-04T09:45:22.1463362Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T09:45:22.1463632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1121, in forward 2025-12-04T09:45:22.1463699Z x = self.dense(features) 2025-12-04T09:45:22.1463702Z 2025-12-04T09:45:22.1463797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1463994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1464054Z res = mod(**inputs) 2025-12-04T09:45:22.1464303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 988, in forward 2025-12-04T09:45:22.1464406Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T09:45:22.1464663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1126, in forward 2025-12-04T09:45:22.1464734Z x = self.decoder(x) 2025-12-04T09:45:22.1464738Z 2025-12-04T09:45:22.1464832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:22.1465014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:22.1465084Z res = mod(**inputs) 2025-12-04T09:45:22.1465330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 994, in forward 2025-12-04T09:45:22.1465406Z lm_loss = self.loss_function( 2025-12-04T09:45:22.1465636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:45:22.1465800Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:45:22.1466055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:45:22.1466258Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:45:22.1466262Z 2025-12-04T09:45:32.3118395Z Compilation time (from dynamo_timed): 16.780476531 2025-12-04T09:45:32.3212359Z pass 2025-12-04T09:45:32.3212759Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:32.3213569Z TIMING: _recursive_pre_grad_passes:0.00742 _recursive_joint_graph_passes:0.66203 _recursive_post_grad_passes:0.06613 async_compile.wait:0.78564 code_gen:9.61784 inductor_compile:10.89621 backend_compile:13.95631 gc:0.00131 entire_frame_compile:16.78048 total_wall_time:16.78048 2025-12-04T09:45:32.3214572Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:7229 | FakeTensor.__torch_dispatch__:4402 | ProxyTorchDispatchMode.__torch_dispatch__:1992 2025-12-04T09:45:32.3215081Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-12-04T09:45:34.8350198Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:45:34.8351108Z import pynvml # type: ignore[import] 2025-12-04T09:45:38.2576340Z 2025-12-04T09:45:39.1036201Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:45:39.1036757Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:45:39.1048156Z cpu eval T5ForConditionalGeneration 2025-12-04T09:45:40.1987294Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:40.6066330Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:41.0740904Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:49.3979611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.3980411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.3980880Z res = mod(**inputs) 2025-12-04T09:45:49.3982086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.3982618Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.3983047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.3983457Z layer_outputs = layer_module( 2025-12-04T09:45:49.3983843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.3984273Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.3984707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.3985129Z return func(*args, **kwargs) 2025-12-04T09:45:49.3985532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.3985955Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.3986370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.3986748Z return func(*args, **kwargs) 2025-12-04T09:45:49.3987109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.3987492Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.3987873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.3988252Z return func(*args, **kwargs) 2025-12-04T09:45:49.3988756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 549, in forward 2025-12-04T09:45:49.3989458Z position_bias = position_bias + causal_mask 2025-12-04T09:45:49.3989627Z 2025-12-04T09:45:49.3989740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.3990195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.3990528Z res = mod(**inputs) 2025-12-04T09:45:49.3990879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.3991264Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.3991639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.3992015Z layer_outputs = layer_module( 2025-12-04T09:45:49.3992372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.3992835Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.3993226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.3993589Z return func(*args, **kwargs) 2025-12-04T09:45:49.3993951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.3994329Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.3994709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.3995067Z return func(*args, **kwargs) 2025-12-04T09:45:49.3995424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.3995862Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.3996240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.3996621Z return func(*args, **kwargs) 2025-12-04T09:45:49.3996974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.3997347Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.3997483Z 2025-12-04T09:45:49.3997591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.3997962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.3998279Z res = mod(**inputs) 2025-12-04T09:45:49.3998617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.3998980Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.3999354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.3999718Z layer_outputs = layer_module( 2025-12-04T09:45:49.4000054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4000416Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4000799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4001175Z return func(*args, **kwargs) 2025-12-04T09:45:49.4001515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4001883Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4002289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4002664Z return func(*args, **kwargs) 2025-12-04T09:45:49.4003038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4003431Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4003820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4004213Z return func(*args, **kwargs) 2025-12-04T09:45:49.4004570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4004947Z key_states = self.k(current_states) 2025-12-04T09:45:49.4005078Z 2025-12-04T09:45:49.4005199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4005605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4005929Z res = mod(**inputs) 2025-12-04T09:45:49.4006294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4006700Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4007097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4007490Z layer_outputs = layer_module( 2025-12-04T09:45:49.4007863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4008244Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4008647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4009059Z return func(*args, **kwargs) 2025-12-04T09:45:49.4009446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4009870Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4010283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4010676Z return func(*args, **kwargs) 2025-12-04T09:45:49.4011043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4011664Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4012083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4012474Z return func(*args, **kwargs) 2025-12-04T09:45:49.4012845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4013239Z value_states = self.v(current_states) 2025-12-04T09:45:49.4013376Z 2025-12-04T09:45:49.4013468Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4013679Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4013917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4014281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4014606Z res = mod(**inputs) 2025-12-04T09:45:49.4014965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4015363Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4015760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4016154Z layer_outputs = layer_module( 2025-12-04T09:45:49.4016509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4016873Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4017254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4017617Z return func(*args, **kwargs) 2025-12-04T09:45:49.4018020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4018433Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4018859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4019250Z return func(*args, **kwargs) 2025-12-04T09:45:49.4019604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4019983Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4020357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4020728Z return func(*args, **kwargs) 2025-12-04T09:45:49.4021101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4021479Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4021632Z 2025-12-04T09:45:49.4021735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4022096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4022422Z res = mod(**inputs) 2025-12-04T09:45:49.4022757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4023158Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4023575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4023992Z layer_outputs = layer_module( 2025-12-04T09:45:49.4024354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4024760Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4025170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4025545Z return func(*args, **kwargs) 2025-12-04T09:45:49.4025899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4026290Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4026691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4027126Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4027540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4027932Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4028072Z 2025-12-04T09:45:49.4028190Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4028570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4028895Z res = mod(**inputs) 2025-12-04T09:45:49.4029249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4029627Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4030025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4030417Z layer_outputs = layer_module( 2025-12-04T09:45:49.4030766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4031126Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4031507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4031881Z return func(*args, **kwargs) 2025-12-04T09:45:49.4032275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4032669Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4033141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4033605Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4034043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4034461Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4034606Z 2025-12-04T09:45:49.4034713Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4035109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4035431Z res = mod(**inputs) 2025-12-04T09:45:49.4035799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4036197Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4036594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4037002Z layer_outputs = layer_module( 2025-12-04T09:45:49.4037367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4037767Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4038179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4038586Z return func(*args, **kwargs) 2025-12-04T09:45:49.4038989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4039408Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4039891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4040316Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4040757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4041450Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4041592Z 2025-12-04T09:45:49.4041708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4042084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4042431Z res = mod(**inputs) 2025-12-04T09:45:49.4042805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4043209Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4043599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4043995Z layer_outputs = layer_module( 2025-12-04T09:45:49.4044367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4044753Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4045165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4045570Z return func(*args, **kwargs) 2025-12-04T09:45:49.4045951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4046348Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4046751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4047144Z return func(*args, **kwargs) 2025-12-04T09:45:49.4047534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4047968Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4048371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4048757Z return func(*args, **kwargs) 2025-12-04T09:45:49.4049120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4049516Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4049659Z 2025-12-04T09:45:49.4049777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4050160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4050489Z res = mod(**inputs) 2025-12-04T09:45:49.4050875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4051375Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4051790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4052209Z layer_outputs = layer_module( 2025-12-04T09:45:49.4052599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4052989Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4053385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4053788Z return func(*args, **kwargs) 2025-12-04T09:45:49.4054207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4054618Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4055033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4055439Z return func(*args, **kwargs) 2025-12-04T09:45:49.4055820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4056230Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4056641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4057042Z return func(*args, **kwargs) 2025-12-04T09:45:49.4057427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4057830Z key_states = self.k(current_states) 2025-12-04T09:45:49.4057983Z 2025-12-04T09:45:49.4058098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4058489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4058841Z res = mod(**inputs) 2025-12-04T09:45:49.4059215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4059635Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4060044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4060450Z layer_outputs = layer_module( 2025-12-04T09:45:49.4060828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4061225Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4061617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4061994Z return func(*args, **kwargs) 2025-12-04T09:45:49.4062368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4062747Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4063141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4063512Z return func(*args, **kwargs) 2025-12-04T09:45:49.4063865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4064242Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4064616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4064989Z return func(*args, **kwargs) 2025-12-04T09:45:49.4065366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4065735Z value_states = self.v(current_states) 2025-12-04T09:45:49.4065879Z 2025-12-04T09:45:49.4065958Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4066173Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4066407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4066756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4067075Z res = mod(**inputs) 2025-12-04T09:45:49.4067413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4067775Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4068137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4068528Z layer_outputs = layer_module( 2025-12-04T09:45:49.4068876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4069233Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4069608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4069978Z return func(*args, **kwargs) 2025-12-04T09:45:49.4070324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4070707Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4071082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4071456Z return func(*args, **kwargs) 2025-12-04T09:45:49.4071809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4072184Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4072880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4073254Z return func(*args, **kwargs) 2025-12-04T09:45:49.4073612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4073987Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4074118Z 2025-12-04T09:45:49.4074227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4074580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4074900Z res = mod(**inputs) 2025-12-04T09:45:49.4075251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4075614Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4075962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4076389Z layer_outputs = layer_module( 2025-12-04T09:45:49.4076744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4077120Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4077489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4077854Z return func(*args, **kwargs) 2025-12-04T09:45:49.4078198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4078569Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4078958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4079401Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4079812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4080183Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4080326Z 2025-12-04T09:45:49.4080437Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4080806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4081108Z res = mod(**inputs) 2025-12-04T09:45:49.4081444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4081802Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4082161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4082558Z layer_outputs = layer_module( 2025-12-04T09:45:49.4082900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4083259Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4083618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4083984Z return func(*args, **kwargs) 2025-12-04T09:45:49.4084330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4084713Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4085079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4085479Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4085878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4086238Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4086377Z 2025-12-04T09:45:49.4086479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4086827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4087145Z res = mod(**inputs) 2025-12-04T09:45:49.4087468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4087830Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4088186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4088547Z layer_outputs = layer_module( 2025-12-04T09:45:49.4088877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4089236Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4089622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4089979Z return func(*args, **kwargs) 2025-12-04T09:45:49.4090353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4090740Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4091099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4091595Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4091987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4092368Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4092504Z 2025-12-04T09:45:49.4092610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4093021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4093367Z res = mod(**inputs) 2025-12-04T09:45:49.4093777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4094169Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4094517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4094880Z layer_outputs = layer_module( 2025-12-04T09:45:49.4095254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4095598Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4095963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4096342Z return func(*args, **kwargs) 2025-12-04T09:45:49.4096690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4097069Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4097437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4097790Z return func(*args, **kwargs) 2025-12-04T09:45:49.4098121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4098478Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4098842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4099186Z return func(*args, **kwargs) 2025-12-04T09:45:49.4099528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4099894Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4100027Z 2025-12-04T09:45:49.4100135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4100492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4100798Z res = mod(**inputs) 2025-12-04T09:45:49.4101130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4101484Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4101842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4102202Z layer_outputs = layer_module( 2025-12-04T09:45:49.4102545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4102885Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4103275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4103689Z return func(*args, **kwargs) 2025-12-04T09:45:49.4104040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4104448Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4104814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4105171Z return func(*args, **kwargs) 2025-12-04T09:45:49.4105502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4105870Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4106237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4106608Z return func(*args, **kwargs) 2025-12-04T09:45:49.4106936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4107285Z key_states = self.k(current_states) 2025-12-04T09:45:49.4107410Z 2025-12-04T09:45:49.4107514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4107842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4108157Z res = mod(**inputs) 2025-12-04T09:45:49.4108475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4108823Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4109156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4109531Z layer_outputs = layer_module( 2025-12-04T09:45:49.4109860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4110206Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4110552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4110902Z return func(*args, **kwargs) 2025-12-04T09:45:49.4111236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4111584Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4111941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4112289Z return func(*args, **kwargs) 2025-12-04T09:45:49.4112627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4112980Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4113339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4113688Z return func(*args, **kwargs) 2025-12-04T09:45:49.4114017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4114383Z value_states = self.v(current_states) 2025-12-04T09:45:49.4114515Z 2025-12-04T09:45:49.4114593Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4114800Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4115019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4115353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4115659Z res = mod(**inputs) 2025-12-04T09:45:49.4115975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4116349Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4116701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4117078Z layer_outputs = layer_module( 2025-12-04T09:45:49.4117411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4117760Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4118127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4118475Z return func(*args, **kwargs) 2025-12-04T09:45:49.4118817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4119182Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4119562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4119911Z return func(*args, **kwargs) 2025-12-04T09:45:49.4120256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4120623Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4120991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4121325Z return func(*args, **kwargs) 2025-12-04T09:45:49.4121655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4121999Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4122138Z 2025-12-04T09:45:49.4122235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4122573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4122871Z res = mod(**inputs) 2025-12-04T09:45:49.4123194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4123537Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4123873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4124264Z layer_outputs = layer_module( 2025-12-04T09:45:49.4124587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4124929Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4125286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4125637Z return func(*args, **kwargs) 2025-12-04T09:45:49.4125971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4126346Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4126717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4127110Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4127497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4127858Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4127983Z 2025-12-04T09:45:49.4128088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4128424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4128731Z res = mod(**inputs) 2025-12-04T09:45:49.4129059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4129431Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4129778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4130150Z layer_outputs = layer_module( 2025-12-04T09:45:49.4130481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4130814Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4131166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4131641Z return func(*args, **kwargs) 2025-12-04T09:45:49.4132028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4132442Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4132891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4133310Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4133712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4134053Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4134187Z 2025-12-04T09:45:49.4134282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4134613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4134910Z res = mod(**inputs) 2025-12-04T09:45:49.4135238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4135611Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4135960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4136303Z layer_outputs = layer_module( 2025-12-04T09:45:49.4136631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4136974Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4137323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4137673Z return func(*args, **kwargs) 2025-12-04T09:45:49.4138004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4138370Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4138721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4139107Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4139488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4139838Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4139965Z 2025-12-04T09:45:49.4140063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4140397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4140701Z res = mod(**inputs) 2025-12-04T09:45:49.4141015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4141366Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4141711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4142059Z layer_outputs = layer_module( 2025-12-04T09:45:49.4142379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4142736Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4143096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4143464Z return func(*args, **kwargs) 2025-12-04T09:45:49.4143803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4144159Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4144524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4144867Z return func(*args, **kwargs) 2025-12-04T09:45:49.4145207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4145572Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4145945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4146304Z return func(*args, **kwargs) 2025-12-04T09:45:49.4146642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4146997Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4147122Z 2025-12-04T09:45:49.4147219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4147561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4147864Z res = mod(**inputs) 2025-12-04T09:45:49.4148187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4148556Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4176297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4176712Z layer_outputs = layer_module( 2025-12-04T09:45:49.4177078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4177456Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4177839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4178213Z return func(*args, **kwargs) 2025-12-04T09:45:49.4178560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4178932Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4179304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4179664Z return func(*args, **kwargs) 2025-12-04T09:45:49.4180010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4180383Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4180752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4181106Z return func(*args, **kwargs) 2025-12-04T09:45:49.4181460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4181833Z key_states = self.k(current_states) 2025-12-04T09:45:49.4181966Z 2025-12-04T09:45:49.4182075Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4182440Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4182756Z res = mod(**inputs) 2025-12-04T09:45:49.4183096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4183598Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4183968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4184364Z layer_outputs = layer_module( 2025-12-04T09:45:49.4184692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4185044Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4185406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4185757Z return func(*args, **kwargs) 2025-12-04T09:45:49.4186089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4186453Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4186870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4187226Z return func(*args, **kwargs) 2025-12-04T09:45:49.4187558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4187924Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4188289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4188630Z return func(*args, **kwargs) 2025-12-04T09:45:49.4188966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4189324Z value_states = self.v(current_states) 2025-12-04T09:45:49.4189494Z 2025-12-04T09:45:49.4189582Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4189782Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4190014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4190369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4190681Z res = mod(**inputs) 2025-12-04T09:45:49.4191029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4191389Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4191746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4192097Z layer_outputs = layer_module( 2025-12-04T09:45:49.4192434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4192802Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4193173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4193542Z return func(*args, **kwargs) 2025-12-04T09:45:49.4193892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4194264Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4194632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4195000Z return func(*args, **kwargs) 2025-12-04T09:45:49.4195355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4195728Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4196094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4196461Z return func(*args, **kwargs) 2025-12-04T09:45:49.4196817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4197200Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4197336Z 2025-12-04T09:45:49.4197440Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4197812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4198128Z res = mod(**inputs) 2025-12-04T09:45:49.4198461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4198827Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4199186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4199546Z layer_outputs = layer_module( 2025-12-04T09:45:49.4199888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4200267Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4200642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4200996Z return func(*args, **kwargs) 2025-12-04T09:45:49.4201342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4201712Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4202076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4202436Z return func(*args, **kwargs) 2025-12-04T09:45:49.4202781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T09:45:49.4203266Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:45:49.4203447Z 2025-12-04T09:45:49.4203549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4203903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4204217Z res = mod(**inputs) 2025-12-04T09:45:49.4204558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4204920Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4205271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4205629Z layer_outputs = layer_module( 2025-12-04T09:45:49.4205965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4206315Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4206693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4207060Z return func(*args, **kwargs) 2025-12-04T09:45:49.4207412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4207796Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4208185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4208599Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4209010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4209380Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4209521Z 2025-12-04T09:45:49.4209624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4209982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4210297Z res = mod(**inputs) 2025-12-04T09:45:49.4210661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4211038Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4211597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4212020Z layer_outputs = layer_module( 2025-12-04T09:45:49.4212420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4212825Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4213225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4213608Z return func(*args, **kwargs) 2025-12-04T09:45:49.4213992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4214386Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4214764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4215179Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4215586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4215966Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4216102Z 2025-12-04T09:45:49.4216207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4216565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4216906Z res = mod(**inputs) 2025-12-04T09:45:49.4217242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4217620Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4217988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4218365Z layer_outputs = layer_module( 2025-12-04T09:45:49.4218706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4219070Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4219447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4219816Z return func(*args, **kwargs) 2025-12-04T09:45:49.4220173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4220564Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4220950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4221357Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4221764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4222145Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4222277Z 2025-12-04T09:45:49.4222387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4222740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4223062Z res = mod(**inputs) 2025-12-04T09:45:49.4223407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4223801Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4224158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4224520Z layer_outputs = layer_module( 2025-12-04T09:45:49.4224877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4225243Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4225608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4225968Z return func(*args, **kwargs) 2025-12-04T09:45:49.4226313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4226669Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4227033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4227394Z return func(*args, **kwargs) 2025-12-04T09:45:49.4227767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4228134Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4228509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4228871Z return func(*args, **kwargs) 2025-12-04T09:45:49.4229215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4229569Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4229707Z 2025-12-04T09:45:49.4229809Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4230158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4230485Z res = mod(**inputs) 2025-12-04T09:45:49.4230818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4231177Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4231531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4231883Z layer_outputs = layer_module( 2025-12-04T09:45:49.4232217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4232568Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4232923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4233280Z return func(*args, **kwargs) 2025-12-04T09:45:49.4233622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4233987Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4234350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4234710Z return func(*args, **kwargs) 2025-12-04T09:45:49.4235054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4235433Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4235789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4236138Z return func(*args, **kwargs) 2025-12-04T09:45:49.4236474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4236821Z key_states = self.k(current_states) 2025-12-04T09:45:49.4236955Z 2025-12-04T09:45:49.4237054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4237392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4237698Z res = mod(**inputs) 2025-12-04T09:45:49.4238033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4238409Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4238760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4239111Z layer_outputs = layer_module( 2025-12-04T09:45:49.4239445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4239797Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4240157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4240505Z return func(*args, **kwargs) 2025-12-04T09:45:49.4240865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4241227Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4241581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4241933Z return func(*args, **kwargs) 2025-12-04T09:45:49.4242267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4242630Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4242984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4243340Z return func(*args, **kwargs) 2025-12-04T09:45:49.4243674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4244047Z value_states = self.v(current_states) 2025-12-04T09:45:49.4244174Z 2025-12-04T09:45:49.4244251Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4244454Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4244678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4245010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4245312Z res = mod(**inputs) 2025-12-04T09:45:49.4245636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4245988Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4246323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4246674Z layer_outputs = layer_module( 2025-12-04T09:45:49.4247003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4247340Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4247700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4248058Z return func(*args, **kwargs) 2025-12-04T09:45:49.4248406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4248763Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4249132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4249490Z return func(*args, **kwargs) 2025-12-04T09:45:49.4249829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4250202Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4250573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4250948Z return func(*args, **kwargs) 2025-12-04T09:45:49.4251378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4251814Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4251952Z 2025-12-04T09:45:49.4252072Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4252456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4252800Z res = mod(**inputs) 2025-12-04T09:45:49.4253164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4253537Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4253899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4254301Z layer_outputs = layer_module( 2025-12-04T09:45:49.4254689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4255093Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4255486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4255881Z return func(*args, **kwargs) 2025-12-04T09:45:49.4256252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4256664Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4257074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4257538Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4257977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4258369Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4258517Z 2025-12-04T09:45:49.4258628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4259010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4259355Z res = mod(**inputs) 2025-12-04T09:45:49.4259715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4260115Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4260504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4260909Z layer_outputs = layer_module( 2025-12-04T09:45:49.4261282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4261672Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4262073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4262462Z return func(*args, **kwargs) 2025-12-04T09:45:49.4262833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4263210Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4263594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4263983Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4264380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4264751Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4264881Z 2025-12-04T09:45:49.4264990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4265354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4265676Z res = mod(**inputs) 2025-12-04T09:45:49.4266032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4266390Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4266746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4267109Z layer_outputs = layer_module( 2025-12-04T09:45:49.4267446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4267793Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4268155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4268532Z return func(*args, **kwargs) 2025-12-04T09:45:49.4268878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4269254Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4269630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4270026Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4270412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4270778Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4270908Z 2025-12-04T09:45:49.4271014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4271373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4271675Z res = mod(**inputs) 2025-12-04T09:45:49.4272011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4272559Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4272918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4273292Z layer_outputs = layer_module( 2025-12-04T09:45:49.4273636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4273999Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4274369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4274743Z return func(*args, **kwargs) 2025-12-04T09:45:49.4275111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4275473Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4275845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4276209Z return func(*args, **kwargs) 2025-12-04T09:45:49.4276550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4276919Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4277292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4277657Z return func(*args, **kwargs) 2025-12-04T09:45:49.4278000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4278353Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4278489Z 2025-12-04T09:45:49.4278593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4279021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4279326Z res = mod(**inputs) 2025-12-04T09:45:49.4279706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4280079Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4280450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4280797Z layer_outputs = layer_module( 2025-12-04T09:45:49.4281131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4281475Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4281824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4282202Z return func(*args, **kwargs) 2025-12-04T09:45:49.4282547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4282904Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4283256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4283605Z return func(*args, **kwargs) 2025-12-04T09:45:49.4283940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4284295Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4284656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4285026Z return func(*args, **kwargs) 2025-12-04T09:45:49.4285361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4285709Z key_states = self.k(current_states) 2025-12-04T09:45:49.4285840Z 2025-12-04T09:45:49.4285936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4286272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4286572Z res = mod(**inputs) 2025-12-04T09:45:49.4286889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4287236Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4287576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4287914Z layer_outputs = layer_module( 2025-12-04T09:45:49.4288244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4288596Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4288958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4289301Z return func(*args, **kwargs) 2025-12-04T09:45:49.4289638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4290001Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4290366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4290728Z return func(*args, **kwargs) 2025-12-04T09:45:49.4291076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4291561Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4291977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4292403Z return func(*args, **kwargs) 2025-12-04T09:45:49.4292758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4293153Z value_states = self.v(current_states) 2025-12-04T09:45:49.4293287Z 2025-12-04T09:45:49.4293367Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4293581Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4293828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4294164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4294475Z res = mod(**inputs) 2025-12-04T09:45:49.4294807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4295169Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4295532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4295894Z layer_outputs = layer_module( 2025-12-04T09:45:49.4296232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4296576Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4296941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4297303Z return func(*args, **kwargs) 2025-12-04T09:45:49.4297647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4298007Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4298397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4298768Z return func(*args, **kwargs) 2025-12-04T09:45:49.4299111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4299485Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4299863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4300228Z return func(*args, **kwargs) 2025-12-04T09:45:49.4300569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4300939Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4301071Z 2025-12-04T09:45:49.4301182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4301530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4301848Z res = mod(**inputs) 2025-12-04T09:45:49.4302189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4302558Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4302912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4303276Z layer_outputs = layer_module( 2025-12-04T09:45:49.4303618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4303974Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4304335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4304697Z return func(*args, **kwargs) 2025-12-04T09:45:49.4305046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4305411Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4305804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4306161Z return func(*args, **kwargs) 2025-12-04T09:45:49.4306525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4306886Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4307257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4307625Z return func(*args, **kwargs) 2025-12-04T09:45:49.4307957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4308319Z key_states = self.k(current_states) 2025-12-04T09:45:49.4308450Z 2025-12-04T09:45:49.4308548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4308904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4309201Z res = mod(**inputs) 2025-12-04T09:45:49.4309528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4309887Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4310228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4310583Z layer_outputs = layer_module( 2025-12-04T09:45:49.4310912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4311258Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4311613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4311981Z return func(*args, **kwargs) 2025-12-04T09:45:49.4312320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4312679Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4313036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4313383Z return func(*args, **kwargs) 2025-12-04T09:45:49.4313717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4314069Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4314429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4314778Z return func(*args, **kwargs) 2025-12-04T09:45:49.4315115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4315461Z value_states = self.v(current_states) 2025-12-04T09:45:49.4315595Z 2025-12-04T09:45:49.4315671Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4315873Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4316089Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4316425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4316733Z res = mod(**inputs) 2025-12-04T09:45:49.4317056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4317399Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4317743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4318096Z layer_outputs = layer_module( 2025-12-04T09:45:49.4318418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4318780Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4319136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4319516Z return func(*args, **kwargs) 2025-12-04T09:45:49.4319848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4320205Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4320564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4320911Z return func(*args, **kwargs) 2025-12-04T09:45:49.4321237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4321594Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4321971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4322315Z return func(*args, **kwargs) 2025-12-04T09:45:49.4322655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4323018Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4323143Z 2025-12-04T09:45:49.4323249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4323594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4323895Z res = mod(**inputs) 2025-12-04T09:45:49.4324218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4324587Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4324928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4325280Z layer_outputs = layer_module( 2025-12-04T09:45:49.4325609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4325950Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4326308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4326662Z return func(*args, **kwargs) 2025-12-04T09:45:49.4326991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4327355Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4327713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4328065Z return func(*args, **kwargs) 2025-12-04T09:45:49.4328402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4328769Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4329140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4329501Z return func(*args, **kwargs) 2025-12-04T09:45:49.4329837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4330198Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4330326Z 2025-12-04T09:45:49.4330435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4330777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4331095Z res = mod(**inputs) 2025-12-04T09:45:49.4331543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4331993Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4332379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4332805Z layer_outputs = layer_module( 2025-12-04T09:45:49.4333182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4333566Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4333965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4334330Z return func(*args, **kwargs) 2025-12-04T09:45:49.4334683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4335054Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4335453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4335821Z return func(*args, **kwargs) 2025-12-04T09:45:49.4336152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4336515Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4336878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4337227Z return func(*args, **kwargs) 2025-12-04T09:45:49.4337562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4337926Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4338070Z 2025-12-04T09:45:49.4338176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4338522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4338822Z res = mod(**inputs) 2025-12-04T09:45:49.4339152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4339512Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4339855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4340207Z layer_outputs = layer_module( 2025-12-04T09:45:49.4340533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4340889Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4341238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4341590Z return func(*args, **kwargs) 2025-12-04T09:45:49.4341927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4342282Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4342644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4342995Z return func(*args, **kwargs) 2025-12-04T09:45:49.4343328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T09:45:49.4343724Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:45:49.4343912Z 2025-12-04T09:45:49.4344010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4344344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4344647Z res = mod(**inputs) 2025-12-04T09:45:49.4344962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4345335Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4345682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4346043Z layer_outputs = layer_module( 2025-12-04T09:45:49.4346368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4346715Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4347076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4347422Z return func(*args, **kwargs) 2025-12-04T09:45:49.4347761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4348129Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4348524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4348921Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4349314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4349670Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4349793Z 2025-12-04T09:45:49.4349889Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4350222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4350524Z res = mod(**inputs) 2025-12-04T09:45:49.4350837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4351191Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4351535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4351885Z layer_outputs = layer_module( 2025-12-04T09:45:49.4352206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4352547Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4352902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4353248Z return func(*args, **kwargs) 2025-12-04T09:45:49.4353573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4353936Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4354299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4354682Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4355064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4355414Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4355542Z 2025-12-04T09:45:49.4355646Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4355974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4356277Z res = mod(**inputs) 2025-12-04T09:45:49.4356599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T09:45:49.4356949Z encoder_outputs = self.encoder( 2025-12-04T09:45:49.4357285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4357640Z layer_outputs = layer_module( 2025-12-04T09:45:49.4357973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4358334Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4358709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4359087Z return func(*args, **kwargs) 2025-12-04T09:45:49.4359439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4359815Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4360194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4360606Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4361003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4361404Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4361550Z 2025-12-04T09:45:49.4361656Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4362021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4362341Z res = mod(**inputs) 2025-12-04T09:45:49.4362704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4363064Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4363414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4363767Z layer_outputs = layer_module( 2025-12-04T09:45:49.4364096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4364468Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4364826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4365189Z return func(*args, **kwargs) 2025-12-04T09:45:49.4365537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4365902Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4366260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4366617Z return func(*args, **kwargs) 2025-12-04T09:45:49.4366958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4367316Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4367684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4368035Z return func(*args, **kwargs) 2025-12-04T09:45:49.4368372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4368723Z key_states = self.k(current_states) 2025-12-04T09:45:49.4368857Z 2025-12-04T09:45:49.4368956Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4369297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4369608Z res = mod(**inputs) 2025-12-04T09:45:49.4369933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4370292Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4370647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4371008Z layer_outputs = layer_module( 2025-12-04T09:45:49.4371459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4371902Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4372490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4372941Z return func(*args, **kwargs) 2025-12-04T09:45:49.4373333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4373731Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4374096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4374461Z return func(*args, **kwargs) 2025-12-04T09:45:49.4374814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4375193Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4375589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4376019Z return func(*args, **kwargs) 2025-12-04T09:45:49.4376364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4376441Z value_states = self.v(current_states) 2025-12-04T09:45:49.4376445Z 2025-12-04T09:45:49.4376530Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4376606Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4376707Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4376907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4376967Z res = mod(**inputs) 2025-12-04T09:45:49.4377243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4377322Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4377552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4377627Z layer_outputs = layer_module( 2025-12-04T09:45:49.4377846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4377920Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4378163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4378229Z return func(*args, **kwargs) 2025-12-04T09:45:49.4378455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4378543Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4378779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4378852Z return func(*args, **kwargs) 2025-12-04T09:45:49.4379080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4379161Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4379403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4379468Z return func(*args, **kwargs) 2025-12-04T09:45:49.4379694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4379777Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4379780Z 2025-12-04T09:45:49.4379880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4380079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4380139Z res = mod(**inputs) 2025-12-04T09:45:49.4380388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4380465Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4380717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4380789Z layer_outputs = layer_module( 2025-12-04T09:45:49.4380996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4381068Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4381297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4381362Z return func(*args, **kwargs) 2025-12-04T09:45:49.4381593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4381689Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4381911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4382031Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4382251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4382324Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4382328Z 2025-12-04T09:45:49.4382431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4382616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4382701Z res = mod(**inputs) 2025-12-04T09:45:49.4382921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4382990Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4383218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4383283Z layer_outputs = layer_module( 2025-12-04T09:45:49.4383492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4383571Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4383793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4383862Z return func(*args, **kwargs) 2025-12-04T09:45:49.4384081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4384170Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4384398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4384507Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4384726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4384809Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4384812Z 2025-12-04T09:45:49.4384906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4385096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4385156Z res = mod(**inputs) 2025-12-04T09:45:49.4385378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4385455Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4385677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4385751Z layer_outputs = layer_module( 2025-12-04T09:45:49.4385974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4386063Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4386299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4386362Z return func(*args, **kwargs) 2025-12-04T09:45:49.4386596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4386689Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4386916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4387033Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4387284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4387360Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4387364Z 2025-12-04T09:45:49.4387465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4387652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4387716Z res = mod(**inputs) 2025-12-04T09:45:49.4387943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4388011Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4388241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4388327Z layer_outputs = layer_module( 2025-12-04T09:45:49.4388535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4388618Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4388841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4388914Z return func(*args, **kwargs) 2025-12-04T09:45:49.4389131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4389205Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4389439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4389502Z return func(*args, **kwargs) 2025-12-04T09:45:49.4389720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4389809Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4390037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4390109Z return func(*args, **kwargs) 2025-12-04T09:45:49.4390331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4390404Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4390408Z 2025-12-04T09:45:49.4390514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4390697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4390764Z res = mod(**inputs) 2025-12-04T09:45:49.4390986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4391054Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4391285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4391357Z layer_outputs = layer_module( 2025-12-04T09:45:49.4391582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4391680Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4391907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4391977Z return func(*args, **kwargs) 2025-12-04T09:45:49.4392203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4392278Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4392514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4392581Z return func(*args, **kwargs) 2025-12-04T09:45:49.4392821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4392911Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4393138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4393210Z return func(*args, **kwargs) 2025-12-04T09:45:49.4393432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4393504Z key_states = self.k(current_states) 2025-12-04T09:45:49.4393508Z 2025-12-04T09:45:49.4393613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4393797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4393882Z res = mod(**inputs) 2025-12-04T09:45:49.4394108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4394177Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4394407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4394476Z layer_outputs = layer_module( 2025-12-04T09:45:49.4394685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4394766Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4394996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4395066Z return func(*args, **kwargs) 2025-12-04T09:45:49.4395287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4395363Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4395601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4395666Z return func(*args, **kwargs) 2025-12-04T09:45:49.4395887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4395971Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4396201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4396271Z return func(*args, **kwargs) 2025-12-04T09:45:49.4396493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4396565Z value_states = self.v(current_states) 2025-12-04T09:45:49.4396568Z 2025-12-04T09:45:49.4396651Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4396724Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4396828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4397031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4397093Z res = mod(**inputs) 2025-12-04T09:45:49.4397323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4397423Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4397653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4397728Z layer_outputs = layer_module( 2025-12-04T09:45:49.4397938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4398016Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4398246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4398334Z return func(*args, **kwargs) 2025-12-04T09:45:49.4398563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4398637Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4398861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4398930Z return func(*args, **kwargs) 2025-12-04T09:45:49.4399148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4399230Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4399457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4399537Z return func(*args, **kwargs) 2025-12-04T09:45:49.4399799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4399872Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4399876Z 2025-12-04T09:45:49.4399979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4400166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4400225Z res = mod(**inputs) 2025-12-04T09:45:49.4400458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4400526Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4400747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4400821Z layer_outputs = layer_module( 2025-12-04T09:45:49.4401030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4401111Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4401337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4401402Z return func(*args, **kwargs) 2025-12-04T09:45:49.4401633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4401708Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4401933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4402003Z return func(*args, **kwargs) 2025-12-04T09:45:49.4402224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4402330Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4402556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4403702Z return func(*args, **kwargs) 2025-12-04T09:45:49.4403948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4404045Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4404049Z 2025-12-04T09:45:49.4404155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4404341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4404399Z res = mod(**inputs) 2025-12-04T09:45:49.4404632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4404702Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4404924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4405018Z layer_outputs = layer_module( 2025-12-04T09:45:49.4405230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4405309Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4405537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4405602Z return func(*args, **kwargs) 2025-12-04T09:45:49.4405831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4405908Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4406131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4406233Z return func(*args, **kwargs) 2025-12-04T09:45:49.4406466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4406554Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4406786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4406849Z return func(*args, **kwargs) 2025-12-04T09:45:49.4407084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4407156Z key_states = self.k(current_states) 2025-12-04T09:45:49.4407159Z 2025-12-04T09:45:49.4407262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4407450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4407510Z res = mod(**inputs) 2025-12-04T09:45:49.4407744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4407812Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4408038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4408112Z layer_outputs = layer_module( 2025-12-04T09:45:49.4408325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4408406Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4408632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4408696Z return func(*args, **kwargs) 2025-12-04T09:45:49.4408927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4409005Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4409238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4409311Z return func(*args, **kwargs) 2025-12-04T09:45:49.4409561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4409667Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4409899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4409964Z return func(*args, **kwargs) 2025-12-04T09:45:49.4410200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4410275Z value_states = self.v(current_states) 2025-12-04T09:45:49.4410279Z 2025-12-04T09:45:49.4410362Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4410438Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4410537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4410749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4410814Z res = mod(**inputs) 2025-12-04T09:45:49.4411048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4411127Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4411426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4411508Z layer_outputs = layer_module( 2025-12-04T09:45:49.4411723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4411799Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4412076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4412176Z return func(*args, **kwargs) 2025-12-04T09:45:49.4412522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4412616Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4412885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4412964Z return func(*args, **kwargs) 2025-12-04T09:45:49.4413225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4413313Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4413588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4413657Z return func(*args, **kwargs) 2025-12-04T09:45:49.4413909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4413993Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4413997Z 2025-12-04T09:45:49.4414100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4414313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4414375Z res = mod(**inputs) 2025-12-04T09:45:49.4414610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4414687Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4414921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4414996Z layer_outputs = layer_module( 2025-12-04T09:45:49.4415212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4415287Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4415552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4415620Z return func(*args, **kwargs) 2025-12-04T09:45:49.4415850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4415965Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4416194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4416314Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4416544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4416618Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4416623Z 2025-12-04T09:45:49.4416728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4416937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4417011Z res = mod(**inputs) 2025-12-04T09:45:49.4417243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4417314Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4417553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4417622Z layer_outputs = layer_module( 2025-12-04T09:45:49.4417838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4417921Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4418157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4418247Z return func(*args, **kwargs) 2025-12-04T09:45:49.4418491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4418579Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4418824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4418936Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4419170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4419257Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4419261Z 2025-12-04T09:45:49.4419360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4419563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4419627Z res = mod(**inputs) 2025-12-04T09:45:49.4419869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4419945Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4420183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4420259Z layer_outputs = layer_module( 2025-12-04T09:45:49.4420480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4420556Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4420805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4420871Z return func(*args, **kwargs) 2025-12-04T09:45:49.4421107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4421203Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4421464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4421585Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4421841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4421915Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4421919Z 2025-12-04T09:45:49.4422025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4422222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4422289Z res = mod(**inputs) 2025-12-04T09:45:49.4422518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4422586Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4422837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4422908Z layer_outputs = layer_module( 2025-12-04T09:45:49.4423122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4423206Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4423440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4423510Z return func(*args, **kwargs) 2025-12-04T09:45:49.4423736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4423814Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4424105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4424170Z return func(*args, **kwargs) 2025-12-04T09:45:49.4424397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4424484Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4424717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4424788Z return func(*args, **kwargs) 2025-12-04T09:45:49.4425013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4425086Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4425089Z 2025-12-04T09:45:49.4425202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4425381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4425447Z res = mod(**inputs) 2025-12-04T09:45:49.4425666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4425731Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4425949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4426013Z layer_outputs = layer_module( 2025-12-04T09:45:49.4426213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4426289Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4426509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4426576Z return func(*args, **kwargs) 2025-12-04T09:45:49.4426794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4426865Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4427108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4427171Z return func(*args, **kwargs) 2025-12-04T09:45:49.4427405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4427488Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4427708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4427774Z return func(*args, **kwargs) 2025-12-04T09:45:49.4427988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4428056Z key_states = self.k(current_states) 2025-12-04T09:45:49.4428060Z 2025-12-04T09:45:49.4428158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4428352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4428419Z res = mod(**inputs) 2025-12-04T09:45:49.4428636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4428703Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4428924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4428987Z layer_outputs = layer_module( 2025-12-04T09:45:49.4429190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4429269Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4429508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4429575Z return func(*args, **kwargs) 2025-12-04T09:45:49.4429797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4429869Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4430101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4430162Z return func(*args, **kwargs) 2025-12-04T09:45:49.4430381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4430462Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4430685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4430754Z return func(*args, **kwargs) 2025-12-04T09:45:49.4430974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4431045Z value_states = self.v(current_states) 2025-12-04T09:45:49.4431049Z 2025-12-04T09:45:49.4431129Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4431198Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4431301Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4431483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4431541Z res = mod(**inputs) 2025-12-04T09:45:49.4431766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4431831Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4432051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4432124Z layer_outputs = layer_module( 2025-12-04T09:45:49.4432332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4432424Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4432653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4432732Z return func(*args, **kwargs) 2025-12-04T09:45:49.4432963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4433037Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4433267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4433338Z return func(*args, **kwargs) 2025-12-04T09:45:49.4433562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4433647Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4433899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4433963Z return func(*args, **kwargs) 2025-12-04T09:45:49.4434195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4434267Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4434271Z 2025-12-04T09:45:49.4434373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4434558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4434617Z res = mod(**inputs) 2025-12-04T09:45:49.4434850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4434935Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4435162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4435238Z layer_outputs = layer_module( 2025-12-04T09:45:49.4435453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4435535Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4435754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4435814Z return func(*args, **kwargs) 2025-12-04T09:45:49.4436037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4436108Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4436327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4436396Z return func(*args, **kwargs) 2025-12-04T09:45:49.4436614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T09:45:49.4436745Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:45:49.4436750Z 2025-12-04T09:45:49.4436842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4437021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4437089Z res = mod(**inputs) 2025-12-04T09:45:49.4437305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4437376Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4437592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4437657Z layer_outputs = layer_module( 2025-12-04T09:45:49.4437867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4437954Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4438177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4438264Z return func(*args, **kwargs) 2025-12-04T09:45:49.4438480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4438561Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4438780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4438839Z return func(*args, **kwargs) 2025-12-04T09:45:49.4439064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4439144Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4439383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4439457Z return func(*args, **kwargs) 2025-12-04T09:45:49.4439673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4439752Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4439756Z 2025-12-04T09:45:49.4439851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4440032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4440099Z res = mod(**inputs) 2025-12-04T09:45:49.4440316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4440404Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4440622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4440686Z layer_outputs = layer_module( 2025-12-04T09:45:49.4440896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4440968Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4441189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4441258Z return func(*args, **kwargs) 2025-12-04T09:45:49.4441471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4441551Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4441772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4441833Z return func(*args, **kwargs) 2025-12-04T09:45:49.4442057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4442134Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4442354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4442422Z return func(*args, **kwargs) 2025-12-04T09:45:49.4442637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4442711Z key_states = self.k(current_states) 2025-12-04T09:45:49.4442715Z 2025-12-04T09:45:49.4442806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4442988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4443056Z res = mod(**inputs) 2025-12-04T09:45:49.4443275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4443373Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4443590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4443672Z layer_outputs = layer_module( 2025-12-04T09:45:49.4443883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4443954Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4444174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4444246Z return func(*args, **kwargs) 2025-12-04T09:45:49.4444466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4444550Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4444788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4444856Z return func(*args, **kwargs) 2025-12-04T09:45:49.4445086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4445166Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4445394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4445466Z return func(*args, **kwargs) 2025-12-04T09:45:49.4445688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4445768Z value_states = self.v(current_states) 2025-12-04T09:45:49.4445788Z 2025-12-04T09:45:49.4445864Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4445937Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4446042Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4446229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4446296Z res = mod(**inputs) 2025-12-04T09:45:49.4446522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4446590Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4446823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4446889Z layer_outputs = layer_module( 2025-12-04T09:45:49.4447099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4447183Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4447408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4447478Z return func(*args, **kwargs) 2025-12-04T09:45:49.4447701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4447779Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4448010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4448074Z return func(*args, **kwargs) 2025-12-04T09:45:49.4448297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4448383Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4448611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4448682Z return func(*args, **kwargs) 2025-12-04T09:45:49.4448906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4448996Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4449000Z 2025-12-04T09:45:49.4449105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4449312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4449379Z res = mod(**inputs) 2025-12-04T09:45:49.4449606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4449674Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4449907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4449978Z layer_outputs = layer_module( 2025-12-04T09:45:49.4450193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4450294Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4450529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4450601Z return func(*args, **kwargs) 2025-12-04T09:45:49.4450829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4450915Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4451155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4451348Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4451605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4451721Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4451725Z 2025-12-04T09:45:49.4451835Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4452052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4452119Z res = mod(**inputs) 2025-12-04T09:45:49.4452372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4452454Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4452705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4452793Z layer_outputs = layer_module( 2025-12-04T09:45:49.4453014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4453094Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4453346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4453414Z return func(*args, **kwargs) 2025-12-04T09:45:49.4453649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4453749Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4453988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4454114Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4454351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4454431Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4454434Z 2025-12-04T09:45:49.4454542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4454733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4454803Z res = mod(**inputs) 2025-12-04T09:45:49.4455057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4455129Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4455391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4455460Z layer_outputs = layer_module( 2025-12-04T09:45:49.4455674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4455755Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4455989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4456063Z return func(*args, **kwargs) 2025-12-04T09:45:49.4456306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4456395Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4456631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4456743Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4456974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4457056Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4457059Z 2025-12-04T09:45:49.4457158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4457355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4457436Z res = mod(**inputs) 2025-12-04T09:45:49.4457668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4457743Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4457973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4458051Z layer_outputs = layer_module( 2025-12-04T09:45:49.4458266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4458340Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4458579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4458646Z return func(*args, **kwargs) 2025-12-04T09:45:49.4458880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4458971Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4459212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4459288Z return func(*args, **kwargs) 2025-12-04T09:45:49.4459524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4459607Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4459853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4459919Z return func(*args, **kwargs) 2025-12-04T09:45:49.4460154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4460242Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4460245Z 2025-12-04T09:45:49.4460349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4460551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4460615Z res = mod(**inputs) 2025-12-04T09:45:49.4460872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4460965Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4461204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4461277Z layer_outputs = layer_module( 2025-12-04T09:45:49.4461495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4461570Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4461813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4461879Z return func(*args, **kwargs) 2025-12-04T09:45:49.4462130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4462219Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4462452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4462524Z return func(*args, **kwargs) 2025-12-04T09:45:49.4462748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4462824Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4463063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4463126Z return func(*args, **kwargs) 2025-12-04T09:45:49.4463349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4463454Z key_states = self.k(current_states) 2025-12-04T09:45:49.4463458Z 2025-12-04T09:45:49.4463558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4463757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4463820Z res = mod(**inputs) 2025-12-04T09:45:49.4464047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4464125Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4464353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4464429Z layer_outputs = layer_module( 2025-12-04T09:45:49.4464641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4464719Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4464957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4465023Z return func(*args, **kwargs) 2025-12-04T09:45:49.4465250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4465335Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4465567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4465640Z return func(*args, **kwargs) 2025-12-04T09:45:49.4465866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4465945Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4466182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4466249Z return func(*args, **kwargs) 2025-12-04T09:45:49.4466490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4466575Z value_states = self.v(current_states) 2025-12-04T09:45:49.4466579Z 2025-12-04T09:45:49.4466673Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4466756Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4466854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4467045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4467115Z res = mod(**inputs) 2025-12-04T09:45:49.4467345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4467420Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4467651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4467718Z layer_outputs = layer_module( 2025-12-04T09:45:49.4467961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4468037Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4468271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4468340Z return func(*args, **kwargs) 2025-12-04T09:45:49.4468567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4468651Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4468884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4468972Z return func(*args, **kwargs) 2025-12-04T09:45:49.4469206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4469286Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4469522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4469593Z return func(*args, **kwargs) 2025-12-04T09:45:49.4469824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4469905Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4469908Z 2025-12-04T09:45:49.4470008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4470203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4470271Z res = mod(**inputs) 2025-12-04T09:45:49.4470505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4470581Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4470816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4470885Z layer_outputs = layer_module( 2025-12-04T09:45:49.4471107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4471183Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4471417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4471490Z return func(*args, **kwargs) 2025-12-04T09:45:49.4471717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4471802Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4472038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4472104Z return func(*args, **kwargs) 2025-12-04T09:45:49.4472550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4472662Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4472904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4472981Z return func(*args, **kwargs) 2025-12-04T09:45:49.4473216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4473296Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4473299Z 2025-12-04T09:45:49.4473400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4473596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4473667Z res = mod(**inputs) 2025-12-04T09:45:49.4473937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4474020Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4474260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4474331Z layer_outputs = layer_module( 2025-12-04T09:45:49.4474565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4474641Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4474886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4474963Z return func(*args, **kwargs) 2025-12-04T09:45:49.4475227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4475315Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4475560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4475625Z return func(*args, **kwargs) 2025-12-04T09:45:49.4475857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4475936Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4476167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4476237Z return func(*args, **kwargs) 2025-12-04T09:45:49.4476466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4476546Z key_states = self.k(current_states) 2025-12-04T09:45:49.4476549Z 2025-12-04T09:45:49.4476649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4476840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4476907Z res = mod(**inputs) 2025-12-04T09:45:49.4477138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4477216Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4477446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4477512Z layer_outputs = layer_module( 2025-12-04T09:45:49.4477733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4477806Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4478038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4478113Z return func(*args, **kwargs) 2025-12-04T09:45:49.4478354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4478440Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4478685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4478750Z return func(*args, **kwargs) 2025-12-04T09:45:49.4478986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4479066Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4479297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4479372Z return func(*args, **kwargs) 2025-12-04T09:45:49.4479612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4479694Z value_states = self.v(current_states) 2025-12-04T09:45:49.4479698Z 2025-12-04T09:45:49.4479774Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4479850Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4479959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4480148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4480210Z res = mod(**inputs) 2025-12-04T09:45:49.4480450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4480519Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4480753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4480840Z layer_outputs = layer_module( 2025-12-04T09:45:49.4481062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4481141Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4481374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4481446Z return func(*args, **kwargs) 2025-12-04T09:45:49.4481677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4481753Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4481993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4482055Z return func(*args, **kwargs) 2025-12-04T09:45:49.4482284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4482370Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4482604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4482674Z return func(*args, **kwargs) 2025-12-04T09:45:49.4482906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4482977Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4482981Z 2025-12-04T09:45:49.4483084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4483273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4483333Z res = mod(**inputs) 2025-12-04T09:45:49.4483574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4483646Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4483897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4483967Z layer_outputs = layer_module( 2025-12-04T09:45:49.4484185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4484288Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4484536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4484610Z return func(*args, **kwargs) 2025-12-04T09:45:49.4484844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4484923Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4485169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4485235Z return func(*args, **kwargs) 2025-12-04T09:45:49.4485493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 652, in forward 2025-12-04T09:45:49.4485638Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T09:45:49.4485644Z 2025-12-04T09:45:49.4485745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4485950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4486012Z res = mod(**inputs) 2025-12-04T09:45:49.4486253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4486334Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4486573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4486668Z layer_outputs = layer_module( 2025-12-04T09:45:49.4486892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4486967Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4487214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4487282Z return func(*args, **kwargs) 2025-12-04T09:45:49.4487516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4487617Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4487857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4487988Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4488238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4488319Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4488323Z 2025-12-04T09:45:49.4488441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4488647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4488715Z res = mod(**inputs) 2025-12-04T09:45:49.4488973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4489046Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4489303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4489377Z layer_outputs = layer_module( 2025-12-04T09:45:49.4489609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4489699Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4489970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4490050Z return func(*args, **kwargs) 2025-12-04T09:45:49.4490296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4490441Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4490700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4490823Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4491080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4491176Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4491180Z 2025-12-04T09:45:49.4491347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4491593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4491664Z res = mod(**inputs) 2025-12-04T09:45:49.4491924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4492013Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4492275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4492357Z layer_outputs = layer_module( 2025-12-04T09:45:49.4492597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4492677Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4492938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4493038Z return func(*args, **kwargs) 2025-12-04T09:45:49.4493297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4493403Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4493658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4493787Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4494052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4494135Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4494138Z 2025-12-04T09:45:49.4494260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4494460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4494523Z res = mod(**inputs) 2025-12-04T09:45:49.4494777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4494846Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4495097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4495166Z layer_outputs = layer_module( 2025-12-04T09:45:49.4495388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4495472Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4495719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4495793Z return func(*args, **kwargs) 2025-12-04T09:45:49.4496036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4496116Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4496390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4496457Z return func(*args, **kwargs) 2025-12-04T09:45:49.4496707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4496796Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4497038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4497111Z return func(*args, **kwargs) 2025-12-04T09:45:49.4497344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4497420Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4497424Z 2025-12-04T09:45:49.4497531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4497742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4497807Z res = mod(**inputs) 2025-12-04T09:45:49.4498052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4498124Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4498369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4498437Z layer_outputs = layer_module( 2025-12-04T09:45:49.4498658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4498745Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4499006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4499079Z return func(*args, **kwargs) 2025-12-04T09:45:49.4499314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4499393Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4499636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4499701Z return func(*args, **kwargs) 2025-12-04T09:45:49.4499932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4500019Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4500257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4500333Z return func(*args, **kwargs) 2025-12-04T09:45:49.4500565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4500640Z key_states = self.k(current_states) 2025-12-04T09:45:49.4500645Z 2025-12-04T09:45:49.4500750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4500940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4501012Z res = mod(**inputs) 2025-12-04T09:45:49.4501249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4501319Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4501559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4501628Z layer_outputs = layer_module( 2025-12-04T09:45:49.4501845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4501932Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4502188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4502265Z return func(*args, **kwargs) 2025-12-04T09:45:49.4502516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4502596Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4502844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4502909Z return func(*args, **kwargs) 2025-12-04T09:45:49.4503145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4503232Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4503476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4503568Z return func(*args, **kwargs) 2025-12-04T09:45:49.4503815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4503893Z value_states = self.v(current_states) 2025-12-04T09:45:49.4503897Z 2025-12-04T09:45:49.4503989Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4504066Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4504166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4504368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4504431Z res = mod(**inputs) 2025-12-04T09:45:49.4504676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4504767Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4505009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4505088Z layer_outputs = layer_module( 2025-12-04T09:45:49.4505314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4505401Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4505648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4505714Z return func(*args, **kwargs) 2025-12-04T09:45:49.4505961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4506040Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4506290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4506364Z return func(*args, **kwargs) 2025-12-04T09:45:49.4506601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4506687Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4506925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4506990Z return func(*args, **kwargs) 2025-12-04T09:45:49.4507228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4507301Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4507305Z 2025-12-04T09:45:49.4507402Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4507601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4507663Z res = mod(**inputs) 2025-12-04T09:45:49.4507910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4507999Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4508233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4508335Z layer_outputs = layer_module( 2025-12-04T09:45:49.4508549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4508629Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4508865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4508930Z return func(*args, **kwargs) 2025-12-04T09:45:49.4509164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4509243Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4509497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4509573Z return func(*args, **kwargs) 2025-12-04T09:45:49.4509803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4509892Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4510180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4510246Z return func(*args, **kwargs) 2025-12-04T09:45:49.4510610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4510683Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4510702Z 2025-12-04T09:45:49.4510810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4511002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4511066Z res = mod(**inputs) 2025-12-04T09:45:49.4511351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4511424Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4511661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4511740Z layer_outputs = layer_module( 2025-12-04T09:45:49.4511960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4512044Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4512284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4512352Z return func(*args, **kwargs) 2025-12-04T09:45:49.4512595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4512677Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4512918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4512992Z return func(*args, **kwargs) 2025-12-04T09:45:49.4513231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4513322Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4513562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4513627Z return func(*args, **kwargs) 2025-12-04T09:45:49.4513869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4513943Z key_states = self.k(current_states) 2025-12-04T09:45:49.4513949Z 2025-12-04T09:45:49.4514076Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4514279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4514357Z res = mod(**inputs) 2025-12-04T09:45:49.4514596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4514666Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4514902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4514982Z layer_outputs = layer_module( 2025-12-04T09:45:49.4515198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4515282Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4515565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4515635Z return func(*args, **kwargs) 2025-12-04T09:45:49.4515877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4515961Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4516203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4516275Z return func(*args, **kwargs) 2025-12-04T09:45:49.4516510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4516601Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4516850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4516934Z return func(*args, **kwargs) 2025-12-04T09:45:49.4517171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4517246Z value_states = self.v(current_states) 2025-12-04T09:45:49.4517250Z 2025-12-04T09:45:49.4517334Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4517412Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4517512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4517715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4517778Z res = mod(**inputs) 2025-12-04T09:45:49.4518041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4518119Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4518354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4518432Z layer_outputs = layer_module( 2025-12-04T09:45:49.4518652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4518729Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4518974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4519042Z return func(*args, **kwargs) 2025-12-04T09:45:49.4519277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4519366Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4519605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4519680Z return func(*args, **kwargs) 2025-12-04T09:45:49.4519917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4520020Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4520270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4520353Z return func(*args, **kwargs) 2025-12-04T09:45:49.4520595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4520678Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4520681Z 2025-12-04T09:45:49.4520783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4520983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4521044Z res = mod(**inputs) 2025-12-04T09:45:49.4521283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4521379Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4521617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4521693Z layer_outputs = layer_module( 2025-12-04T09:45:49.4521914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4521989Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4522235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4522301Z return func(*args, **kwargs) 2025-12-04T09:45:49.4522532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4522650Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4522885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4523010Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4523243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4523322Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4523325Z 2025-12-04T09:45:49.4523433Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4523625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4523692Z res = mod(**inputs) 2025-12-04T09:45:49.4523928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4523998Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4524245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4524315Z layer_outputs = layer_module( 2025-12-04T09:45:49.4524538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4524624Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4524864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4524935Z return func(*args, **kwargs) 2025-12-04T09:45:49.4525167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4525256Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4525498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4525614Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4525863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4525951Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4525954Z 2025-12-04T09:45:49.4526055Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4526274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4526337Z res = mod(**inputs) 2025-12-04T09:45:49.4526574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4526653Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4526886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4526964Z layer_outputs = layer_module( 2025-12-04T09:45:49.4527192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4527285Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4527534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4527602Z return func(*args, **kwargs) 2025-12-04T09:45:49.4527835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4527932Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4528163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4528282Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4528513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4528610Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4528614Z 2025-12-04T09:45:49.4528725Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4528922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4528993Z res = mod(**inputs) 2025-12-04T09:45:49.4529230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4529300Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4529541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4529612Z layer_outputs = layer_module( 2025-12-04T09:45:49.4529832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4529918Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4530169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4530245Z return func(*args, **kwargs) 2025-12-04T09:45:49.4530478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4530568Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4530810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 344, in forward 2025-12-04T09:45:49.4530940Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-12-04T09:45:49.4530944Z 2025-12-04T09:45:49.4531053Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4531387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4531486Z res = mod(**inputs) 2025-12-04T09:45:49.4531759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4531839Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4532139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4532243Z layer_outputs = layer_module( 2025-12-04T09:45:49.4532497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4532591Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4532859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4532942Z return func(*args, **kwargs) 2025-12-04T09:45:49.4533190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4533275Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4533546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4533633Z return func(*args, **kwargs) 2025-12-04T09:45:49.4533890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4533986Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4534246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4534319Z return func(*args, **kwargs) 2025-12-04T09:45:49.4534582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4534664Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4534668Z 2025-12-04T09:45:49.4534804Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4535015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4535084Z res = mod(**inputs) 2025-12-04T09:45:49.4535350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4535426Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4535684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4535771Z layer_outputs = layer_module( 2025-12-04T09:45:49.4536019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4536110Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4536378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4536452Z return func(*args, **kwargs) 2025-12-04T09:45:49.4536718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4536805Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4537064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4537145Z return func(*args, **kwargs) 2025-12-04T09:45:49.4537409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4537504Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4537774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4537845Z return func(*args, **kwargs) 2025-12-04T09:45:49.4538106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4538189Z key_states = self.k(current_states) 2025-12-04T09:45:49.4538193Z 2025-12-04T09:45:49.4538311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4538541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4538610Z res = mod(**inputs) 2025-12-04T09:45:49.4538902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4538978Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4539242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4539325Z layer_outputs = layer_module( 2025-12-04T09:45:49.4539560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4539650Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4539934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4540010Z return func(*args, **kwargs) 2025-12-04T09:45:49.4540274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4540360Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4540632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4540708Z return func(*args, **kwargs) 2025-12-04T09:45:49.4540969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4541061Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4541330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4541423Z return func(*args, **kwargs) 2025-12-04T09:45:49.4541688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4541771Z value_states = self.v(current_states) 2025-12-04T09:45:49.4541775Z 2025-12-04T09:45:49.4541865Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4541951Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4542067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4542262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4542323Z res = mod(**inputs) 2025-12-04T09:45:49.4542554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4542631Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4542860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4542934Z layer_outputs = layer_module( 2025-12-04T09:45:49.4543151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4543224Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4543464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4543528Z return func(*args, **kwargs) 2025-12-04T09:45:49.4543756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T09:45:49.4543838Z self_attention_outputs = self.layer[0]( 2025-12-04T09:45:49.4544073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4544147Z return func(*args, **kwargs) 2025-12-04T09:45:49.4544373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T09:45:49.4544452Z attention_output = self.SelfAttention( 2025-12-04T09:45:49.4544707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4544789Z return func(*args, **kwargs) 2025-12-04T09:45:49.4545015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4545096Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4545099Z 2025-12-04T09:45:49.4545197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4545393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4545453Z res = mod(**inputs) 2025-12-04T09:45:49.4545684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4545761Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4546005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4546082Z layer_outputs = layer_module( 2025-12-04T09:45:49.4546297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4546372Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4546608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4546674Z return func(*args, **kwargs) 2025-12-04T09:45:49.4546899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4546985Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4547235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4547307Z return func(*args, **kwargs) 2025-12-04T09:45:49.4547535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4547616Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4547854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4547916Z return func(*args, **kwargs) 2025-12-04T09:45:49.4548141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T09:45:49.4548221Z query_states = self.q(hidden_states) 2025-12-04T09:45:49.4548225Z 2025-12-04T09:45:49.4548322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4548519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4548578Z res = mod(**inputs) 2025-12-04T09:45:49.4548807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4548883Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4549109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4549184Z layer_outputs = layer_module( 2025-12-04T09:45:49.4549398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4549472Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4549711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4549776Z return func(*args, **kwargs) 2025-12-04T09:45:49.4550004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4550091Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4550343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4550416Z return func(*args, **kwargs) 2025-12-04T09:45:49.4550661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4550743Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4550981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4551045Z return func(*args, **kwargs) 2025-12-04T09:45:49.4551270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T09:45:49.4551351Z key_states = self.k(current_states) 2025-12-04T09:45:49.4551355Z 2025-12-04T09:45:49.4551451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4551665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4551728Z res = mod(**inputs) 2025-12-04T09:45:49.4551954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4552034Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4552262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4552336Z layer_outputs = layer_module( 2025-12-04T09:45:49.4552550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4552623Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4552890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4552956Z return func(*args, **kwargs) 2025-12-04T09:45:49.4553182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4553265Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4553498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4553568Z return func(*args, **kwargs) 2025-12-04T09:45:49.4553794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4553872Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4554112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4554176Z return func(*args, **kwargs) 2025-12-04T09:45:49.4554403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T09:45:49.4554483Z value_states = self.v(current_states) 2025-12-04T09:45:49.4554487Z 2025-12-04T09:45:49.4554562Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4554643Z cudagraph partition due to non gpu ops 2025-12-04T09:45:49.4554742Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4554931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4554997Z res = mod(**inputs) 2025-12-04T09:45:49.4555225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4555300Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4555527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4555595Z layer_outputs = layer_module( 2025-12-04T09:45:49.4555820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4555911Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4556145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4556232Z return func(*args, **kwargs) 2025-12-04T09:45:49.4556459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T09:45:49.4556544Z cross_attention_outputs = self.layer[1]( 2025-12-04T09:45:49.4556776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4556840Z return func(*args, **kwargs) 2025-12-04T09:45:49.4557074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T09:45:49.4557171Z attention_output = self.EncDecAttention( 2025-12-04T09:45:49.4557410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4557483Z return func(*args, **kwargs) 2025-12-04T09:45:49.4557712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T09:45:49.4557792Z attn_output = self.o(attn_output) 2025-12-04T09:45:49.4557795Z 2025-12-04T09:45:49.4557896Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4558088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4558158Z res = mod(**inputs) 2025-12-04T09:45:49.4558395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4558492Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4558732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4558804Z layer_outputs = layer_module( 2025-12-04T09:45:49.4559035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4559112Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4559353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4559425Z return func(*args, **kwargs) 2025-12-04T09:45:49.4559658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4559756Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4559999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4560114Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4560350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T09:45:49.4560426Z hidden_states = self.wi(hidden_states) 2025-12-04T09:45:49.4560431Z 2025-12-04T09:45:49.4560540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4560735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4560795Z res = mod(**inputs) 2025-12-04T09:45:49.4561041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4561118Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4561352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4561431Z layer_outputs = layer_module( 2025-12-04T09:45:49.4561668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4561753Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4561993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4562078Z return func(*args, **kwargs) 2025-12-04T09:45:49.4562317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4562406Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4562640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4562764Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4563006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T09:45:49.4563106Z hidden_states = self.act(hidden_states) 2025-12-04T09:45:49.4563110Z 2025-12-04T09:45:49.4563216Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4563413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4563485Z res = mod(**inputs) 2025-12-04T09:45:49.4563721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T09:45:49.4563798Z decoder_outputs = self.decoder( 2025-12-04T09:45:49.4564032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T09:45:49.4564101Z layer_outputs = layer_module( 2025-12-04T09:45:49.4564326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:45:49.4564420Z return super().__call__(*args, **kwargs) 2025-12-04T09:45:49.4564664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:45:49.4564741Z return func(*args, **kwargs) 2025-12-04T09:45:49.4564973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T09:45:49.4565069Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T09:45:49.4565301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T09:45:49.4565413Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T09:45:49.4565652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T09:45:49.4565731Z hidden_states = self.wo(hidden_states) 2025-12-04T09:45:49.4565734Z 2025-12-04T09:45:49.4565839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4566046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4566111Z res = mod(**inputs) 2025-12-04T09:45:49.4566369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1793, in forward 2025-12-04T09:45:49.4566459Z lm_logits = self.lm_head(sequence_output) 2025-12-04T09:45:49.4566463Z 2025-12-04T09:45:49.4566567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:45:49.4566780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:45:49.4566846Z res = mod(**inputs) 2025-12-04T09:45:49.4567101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1800, in forward 2025-12-04T09:45:49.4567251Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-12-04T09:45:49.4567255Z 2025-12-04T09:45:59.8306349Z Compilation time (from dynamo_timed): 17.937261795 2025-12-04T09:45:59.8454949Z pass 2025-12-04T09:45:59.8456533Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:45:59.8457583Z TIMING: _recursive_pre_grad_passes:0.01083 _recursive_joint_graph_passes:0.59018 _recursive_post_grad_passes:0.06 async_compile.wait:0.7763 code_gen:9.87663 inductor_compile:11.13622 backend_compile:15.00048 gc:0.00023 entire_frame_compile:17.93726 total_wall_time:17.93726 2025-12-04T09:45:59.8459859Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:11485 | FakeTensor.__torch_dispatch__:4072 | ProxyTorchDispatchMode.__torch_dispatch__:3376 2025-12-04T09:45:59.8466157Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-12-04T09:46:02.3637078Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:46:02.3639018Z import pynvml # type: ignore[import] 2025-12-04T09:46:05.7335258Z 2025-12-04T09:46:06.5853552Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:46:06.5853884Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:46:06.5864774Z cpu eval T5Small 2025-12-04T09:46:07.7006716Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:08.1099001Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:08.5249602Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:23.4888500Z Compilation time (from dynamo_timed): 14.170831869 2025-12-04T09:46:23.5027804Z pass 2025-12-04T09:46:23.5028222Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:23.5028997Z TIMING: _recursive_pre_grad_passes:0.01069 _recursive_joint_graph_passes:0.59684 async_compile.wait:0.00393 inductor_compile:7.32872 backend_compile:11.20082 gc:0.00123 entire_frame_compile:14.17083 total_wall_time:14.17083 2025-12-04T09:46:23.5029842Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:9926 | FakeTensor.__torch_dispatch__:17 | ProxyTorchDispatchMode.__torch_dispatch__:3368 2025-12-04T09:46:23.5030364Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-12-04T09:46:25.8139656Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:46:25.8140609Z import pynvml # type: ignore[import] 2025-12-04T09:46:29.1863680Z 2025-12-04T09:46:31.2763049Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:46:31.2766280Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:46:31.2783928Z cpu eval TrOCRForCausalLM 2025-12-04T09:46:31.4435237Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-12-04T09:46:31.4758689Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:31.7769986Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:32.0279703Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:39.3140786Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3141664Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3141992Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3142255Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3142496Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3142733Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3142976Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3143556Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3143796Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3144088Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3144322Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3144552Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3144856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3145308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3145693Z res = mod(**inputs) 2025-12-04T09:46:39.3146147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3146615Z outputs = self.model.decoder( 2025-12-04T09:46:39.3147105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3147563Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3147975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3148409Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3148861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3149293Z return func(*args, **kwargs) 2025-12-04T09:46:39.3149718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3150181Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3150642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3151117Z return func(*args, **kwargs) 2025-12-04T09:46:39.3151534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3152023Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3152223Z 2025-12-04T09:46:39.3152344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3152759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3153122Z res = mod(**inputs) 2025-12-04T09:46:39.3153534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3153994Z outputs = self.model.decoder( 2025-12-04T09:46:39.3154448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3154887Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3155285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3155706Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3156138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3156561Z return func(*args, **kwargs) 2025-12-04T09:46:39.3156986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3157466Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3157898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3158393Z return func(*args, **kwargs) 2025-12-04T09:46:39.3158805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3159238Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3159388Z 2025-12-04T09:46:39.3159528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3159959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3160341Z res = mod(**inputs) 2025-12-04T09:46:39.3160725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3161168Z outputs = self.model.decoder( 2025-12-04T09:46:39.3161595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3162036Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3162419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3162831Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3163291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3163723Z return func(*args, **kwargs) 2025-12-04T09:46:39.3164143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3164592Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3165032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3165434Z return func(*args, **kwargs) 2025-12-04T09:46:39.3165848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3166281Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3166463Z 2025-12-04T09:46:39.3166555Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3166776Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3166999Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3167250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3167629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3167988Z res = mod(**inputs) 2025-12-04T09:46:39.3168382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3168841Z outputs = self.model.decoder( 2025-12-04T09:46:39.3169274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3169720Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3170127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3170534Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3170961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3171387Z return func(*args, **kwargs) 2025-12-04T09:46:39.3172005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3172695Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3173171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3173593Z return func(*args, **kwargs) 2025-12-04T09:46:39.3174007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3174467Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3174626Z 2025-12-04T09:46:39.3174750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3175152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3175544Z res = mod(**inputs) 2025-12-04T09:46:39.3175945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3176436Z outputs = self.model.decoder( 2025-12-04T09:46:39.3176847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3177267Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3177656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3178057Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3178470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3178888Z return func(*args, **kwargs) 2025-12-04T09:46:39.3179336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3179809Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3180007Z 2025-12-04T09:46:39.3180121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3180522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3180879Z res = mod(**inputs) 2025-12-04T09:46:39.3181270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3181703Z outputs = self.model.decoder( 2025-12-04T09:46:39.3182123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3182585Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3182964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3183364Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3183777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3184184Z return func(*args, **kwargs) 2025-12-04T09:46:39.3184590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3185060Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3185484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3185861Z return self.act(input) 2025-12-04T09:46:39.3185989Z 2025-12-04T09:46:39.3186104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3186497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3186840Z res = mod(**inputs) 2025-12-04T09:46:39.3187235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3187648Z outputs = self.model.decoder( 2025-12-04T09:46:39.3188050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3188449Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3188818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3189201Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3189599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3189987Z return func(*args, **kwargs) 2025-12-04T09:46:39.3190383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3190822Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3190974Z 2025-12-04T09:46:39.3191087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3191514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3191856Z res = mod(**inputs) 2025-12-04T09:46:39.3192242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3192662Z outputs = self.model.decoder( 2025-12-04T09:46:39.3193065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3193484Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3193862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3194283Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3194699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3195101Z return func(*args, **kwargs) 2025-12-04T09:46:39.3195499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3195956Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3196380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3196784Z return func(*args, **kwargs) 2025-12-04T09:46:39.3197166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3197648Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3197819Z 2025-12-04T09:46:39.3197936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3198312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3198657Z res = mod(**inputs) 2025-12-04T09:46:39.3199040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3199461Z outputs = self.model.decoder( 2025-12-04T09:46:39.3199856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3200267Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3200638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3201017Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3201420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3201813Z return func(*args, **kwargs) 2025-12-04T09:46:39.3202213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3202654Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3203086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3203488Z return func(*args, **kwargs) 2025-12-04T09:46:39.3203890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3204303Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3204451Z 2025-12-04T09:46:39.3204563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3204941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3205273Z res = mod(**inputs) 2025-12-04T09:46:39.3205687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3206130Z outputs = self.model.decoder( 2025-12-04T09:46:39.3206541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3206948Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3207319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3207713Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3208111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3208510Z return func(*args, **kwargs) 2025-12-04T09:46:39.3208965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3209409Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3209833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3210240Z return func(*args, **kwargs) 2025-12-04T09:46:39.3210634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3211069Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3211217Z 2025-12-04T09:46:39.3211304Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3211646Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3211889Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3212171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3212572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3212924Z res = mod(**inputs) 2025-12-04T09:46:39.3213312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3213738Z outputs = self.model.decoder( 2025-12-04T09:46:39.3214150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3214562Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3214933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3215326Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3215737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3216148Z return func(*args, **kwargs) 2025-12-04T09:46:39.3216546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3216997Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3217435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3217833Z return func(*args, **kwargs) 2025-12-04T09:46:39.3218242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3218672Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3218822Z 2025-12-04T09:46:39.3218944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3219326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3219684Z res = mod(**inputs) 2025-12-04T09:46:39.3220077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3220511Z outputs = self.model.decoder( 2025-12-04T09:46:39.3220925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3221367Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3221742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3222128Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3222535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3222946Z return func(*args, **kwargs) 2025-12-04T09:46:39.3223345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3223804Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3223995Z 2025-12-04T09:46:39.3224128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3224520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3224864Z res = mod(**inputs) 2025-12-04T09:46:39.3225254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3225676Z outputs = self.model.decoder( 2025-12-04T09:46:39.3226075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3226453Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3226805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3227193Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3227576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3227958Z return func(*args, **kwargs) 2025-12-04T09:46:39.3228345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3228784Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3229173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3229531Z return self.act(input) 2025-12-04T09:46:39.3229645Z 2025-12-04T09:46:39.3229759Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3230125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3230444Z res = mod(**inputs) 2025-12-04T09:46:39.3230813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3231210Z outputs = self.model.decoder( 2025-12-04T09:46:39.3231596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3231993Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3232440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3232814Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3233185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3233554Z return func(*args, **kwargs) 2025-12-04T09:46:39.3233972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3234373Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3234522Z 2025-12-04T09:46:39.3234629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3235016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3235345Z res = mod(**inputs) 2025-12-04T09:46:39.3235723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3236115Z outputs = self.model.decoder( 2025-12-04T09:46:39.3236498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3236883Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3237230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3237595Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3237977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3238362Z return func(*args, **kwargs) 2025-12-04T09:46:39.3238744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3239164Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3239568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3239940Z return func(*args, **kwargs) 2025-12-04T09:46:39.3240315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3240760Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3240924Z 2025-12-04T09:46:39.3241033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3241408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3241734Z res = mod(**inputs) 2025-12-04T09:46:39.3242094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3242477Z outputs = self.model.decoder( 2025-12-04T09:46:39.3242860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3243251Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3243599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3243959Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3244339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3244717Z return func(*args, **kwargs) 2025-12-04T09:46:39.3245089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3245506Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3245908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3246283Z return func(*args, **kwargs) 2025-12-04T09:46:39.3246652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3247048Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3247184Z 2025-12-04T09:46:39.3247297Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3247649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3247970Z res = mod(**inputs) 2025-12-04T09:46:39.3248333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3248723Z outputs = self.model.decoder( 2025-12-04T09:46:39.3249132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3249536Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3249882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3250243Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3250638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3251032Z return func(*args, **kwargs) 2025-12-04T09:46:39.3251419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3251931Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3252402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3252818Z return func(*args, **kwargs) 2025-12-04T09:46:39.3253222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3253654Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3253815Z 2025-12-04T09:46:39.3253894Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3254110Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3254314Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3254552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3254912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3255259Z res = mod(**inputs) 2025-12-04T09:46:39.3255622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3256038Z outputs = self.model.decoder( 2025-12-04T09:46:39.3256437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3256816Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3257167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3257528Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3257902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3258268Z return func(*args, **kwargs) 2025-12-04T09:46:39.3258645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3259049Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3259429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3259791Z return func(*args, **kwargs) 2025-12-04T09:46:39.3260154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3260544Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3260676Z 2025-12-04T09:46:39.3260777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3261125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3261433Z res = mod(**inputs) 2025-12-04T09:46:39.3261776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3262148Z outputs = self.model.decoder( 2025-12-04T09:46:39.3262518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3262893Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3263241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3263615Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3263982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3264343Z return func(*args, **kwargs) 2025-12-04T09:46:39.3264698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3265120Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3265285Z 2025-12-04T09:46:39.3265395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3265747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3266076Z res = mod(**inputs) 2025-12-04T09:46:39.3266429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3266805Z outputs = self.model.decoder( 2025-12-04T09:46:39.3267168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3267542Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3267882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3268234Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3268598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3268984Z return func(*args, **kwargs) 2025-12-04T09:46:39.3269348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3269761Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3270140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3270474Z return self.act(input) 2025-12-04T09:46:39.3270583Z 2025-12-04T09:46:39.3270692Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3271034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3271354Z res = mod(**inputs) 2025-12-04T09:46:39.3271705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3272076Z outputs = self.model.decoder( 2025-12-04T09:46:39.3272634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3273025Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3273373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3273722Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3274100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3274468Z return func(*args, **kwargs) 2025-12-04T09:46:39.3274834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3275213Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3275361Z 2025-12-04T09:46:39.3275466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3275819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3276126Z res = mod(**inputs) 2025-12-04T09:46:39.3276534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3276913Z outputs = self.model.decoder( 2025-12-04T09:46:39.3277278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3277682Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3278023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3278380Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3278738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3279102Z return func(*args, **kwargs) 2025-12-04T09:46:39.3279467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3279892Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3280278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3280645Z return func(*args, **kwargs) 2025-12-04T09:46:39.3281010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3281423Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3281579Z 2025-12-04T09:46:39.3281679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3282026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3282337Z res = mod(**inputs) 2025-12-04T09:46:39.3283494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3283872Z outputs = self.model.decoder( 2025-12-04T09:46:39.3284248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3284614Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3284940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3285285Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3285644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3285987Z return func(*args, **kwargs) 2025-12-04T09:46:39.3286340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3286731Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3287112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3287457Z return func(*args, **kwargs) 2025-12-04T09:46:39.3287811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3288189Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3288315Z 2025-12-04T09:46:39.3288421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3288751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3289058Z res = mod(**inputs) 2025-12-04T09:46:39.3289402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3289772Z outputs = self.model.decoder( 2025-12-04T09:46:39.3290141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3290517Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3290880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3291240Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3291764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3292187Z return func(*args, **kwargs) 2025-12-04T09:46:39.3292586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3293051Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3293465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3293847Z return func(*args, **kwargs) 2025-12-04T09:46:39.3294244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3294658Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3294801Z 2025-12-04T09:46:39.3294888Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3295119Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3295335Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3295581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3295958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3296291Z res = mod(**inputs) 2025-12-04T09:46:39.3296648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3297037Z outputs = self.model.decoder( 2025-12-04T09:46:39.3297456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3297844Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3298189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3298552Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3298921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3299295Z return func(*args, **kwargs) 2025-12-04T09:46:39.3299667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3300077Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3300467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3300842Z return func(*args, **kwargs) 2025-12-04T09:46:39.3301217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3301609Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3301754Z 2025-12-04T09:46:39.3301857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3302213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3302531Z res = mod(**inputs) 2025-12-04T09:46:39.3302885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3303266Z outputs = self.model.decoder( 2025-12-04T09:46:39.3303643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3304029Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3304370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3304736Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3305133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3305517Z return func(*args, **kwargs) 2025-12-04T09:46:39.3305891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3306324Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3306497Z 2025-12-04T09:46:39.3306608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3306960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3307281Z res = mod(**inputs) 2025-12-04T09:46:39.3307643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3308041Z outputs = self.model.decoder( 2025-12-04T09:46:39.3308428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3308816Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3309169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3309523Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3309902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3310273Z return func(*args, **kwargs) 2025-12-04T09:46:39.3310636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3311091Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3311481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3311827Z return self.act(input) 2025-12-04T09:46:39.3311939Z 2025-12-04T09:46:39.3312040Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3312402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3312725Z res = mod(**inputs) 2025-12-04T09:46:39.3313085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3313464Z outputs = self.model.decoder( 2025-12-04T09:46:39.3313843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3314233Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3314567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3314925Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3315296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3315657Z return func(*args, **kwargs) 2025-12-04T09:46:39.3316021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3316414Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3316550Z 2025-12-04T09:46:39.3316661Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3317010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3317331Z res = mod(**inputs) 2025-12-04T09:46:39.3317686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3318075Z outputs = self.model.decoder( 2025-12-04T09:46:39.3318472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3318862Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3319209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3319594Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3319964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3320335Z return func(*args, **kwargs) 2025-12-04T09:46:39.3320705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3321175Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3321563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3321961Z return func(*args, **kwargs) 2025-12-04T09:46:39.3322338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3322755Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3322929Z 2025-12-04T09:46:39.3323032Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3323390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3323715Z res = mod(**inputs) 2025-12-04T09:46:39.3324063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3324450Z outputs = self.model.decoder( 2025-12-04T09:46:39.3324817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3325209Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3325551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3325904Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3326273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3326626Z return func(*args, **kwargs) 2025-12-04T09:46:39.3326988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3327389Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3327770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3328132Z return func(*args, **kwargs) 2025-12-04T09:46:39.3328497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3328891Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3329025Z 2025-12-04T09:46:39.3329128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3329485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3329807Z res = mod(**inputs) 2025-12-04T09:46:39.3330156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3330547Z outputs = self.model.decoder( 2025-12-04T09:46:39.3330922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3331308Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3331740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3332122Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3332538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3332911Z return func(*args, **kwargs) 2025-12-04T09:46:39.3333489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3333905Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3334307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3334671Z return func(*args, **kwargs) 2025-12-04T09:46:39.3335047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3335462Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3335611Z 2025-12-04T09:46:39.3335705Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3335944Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3336173Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3336408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3336770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3337113Z res = mod(**inputs) 2025-12-04T09:46:39.3337490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3337879Z outputs = self.model.decoder( 2025-12-04T09:46:39.3338257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3338648Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3339024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3339391Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3339766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3340140Z return func(*args, **kwargs) 2025-12-04T09:46:39.3340517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3340926Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3341323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3341698Z return func(*args, **kwargs) 2025-12-04T09:46:39.3342073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3342487Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3342640Z 2025-12-04T09:46:39.3342751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3343109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3343422Z res = mod(**inputs) 2025-12-04T09:46:39.3343780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3344165Z outputs = self.model.decoder( 2025-12-04T09:46:39.3344542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3344924Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3345274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3345635Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3346006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3346376Z return func(*args, **kwargs) 2025-12-04T09:46:39.3346775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3347228Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3347401Z 2025-12-04T09:46:39.3347504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3347866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3348188Z res = mod(**inputs) 2025-12-04T09:46:39.3348543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3348922Z outputs = self.model.decoder( 2025-12-04T09:46:39.3349298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3349685Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3350061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3350423Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3350802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3351171Z return func(*args, **kwargs) 2025-12-04T09:46:39.3351534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3351961Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3352337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3352692Z return self.act(input) 2025-12-04T09:46:39.3352798Z 2025-12-04T09:46:39.3352899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3353256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3353574Z res = mod(**inputs) 2025-12-04T09:46:39.3353920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3354314Z outputs = self.model.decoder( 2025-12-04T09:46:39.3354707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3355091Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3355427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3355788Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3356162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3356527Z return func(*args, **kwargs) 2025-12-04T09:46:39.3356912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3357315Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3357458Z 2025-12-04T09:46:39.3357569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3357926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3358256Z res = mod(**inputs) 2025-12-04T09:46:39.3358626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3359037Z outputs = self.model.decoder( 2025-12-04T09:46:39.3359443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3359857Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3360259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3360620Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3360997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3361400Z return func(*args, **kwargs) 2025-12-04T09:46:39.3361797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3362223Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3362629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3363035Z return func(*args, **kwargs) 2025-12-04T09:46:39.3363428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3363906Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3364083Z 2025-12-04T09:46:39.3364189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3364549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3364870Z res = mod(**inputs) 2025-12-04T09:46:39.3365228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3365622Z outputs = self.model.decoder( 2025-12-04T09:46:39.3366015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3366437Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3366805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3367224Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3367632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3368042Z return func(*args, **kwargs) 2025-12-04T09:46:39.3368443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3368892Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3369314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3369713Z return func(*args, **kwargs) 2025-12-04T09:46:39.3370105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3370527Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3370678Z 2025-12-04T09:46:39.3370786Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3371165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3371599Z res = mod(**inputs) 2025-12-04T09:46:39.3371993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3372657Z outputs = self.model.decoder( 2025-12-04T09:46:39.3373065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3373487Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3373866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3374256Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3374669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3375068Z return func(*args, **kwargs) 2025-12-04T09:46:39.3375532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3375989Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3376445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3376841Z return func(*args, **kwargs) 2025-12-04T09:46:39.3377243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3377691Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3377842Z 2025-12-04T09:46:39.3377929Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3378159Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3378386Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3378642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3379045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3379389Z res = mod(**inputs) 2025-12-04T09:46:39.3379780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3380163Z outputs = self.model.decoder( 2025-12-04T09:46:39.3380542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3380925Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3381273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3381630Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3382047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3382424Z return func(*args, **kwargs) 2025-12-04T09:46:39.3382902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3383327Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3383731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3384105Z return func(*args, **kwargs) 2025-12-04T09:46:39.3384473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3384870Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3385004Z 2025-12-04T09:46:39.3385118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3385480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3385793Z res = mod(**inputs) 2025-12-04T09:46:39.3386154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3386544Z outputs = self.model.decoder( 2025-12-04T09:46:39.3386911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3387299Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3387648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3388008Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3388377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3388805Z return func(*args, **kwargs) 2025-12-04T09:46:39.3389176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3389600Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3389809Z 2025-12-04T09:46:39.3389916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3390270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3390613Z res = mod(**inputs) 2025-12-04T09:46:39.3390969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3391362Z outputs = self.model.decoder( 2025-12-04T09:46:39.3391743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3392134Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3392480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3392848Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3393255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3393620Z return func(*args, **kwargs) 2025-12-04T09:46:39.3393994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3394423Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3394809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3395147Z return self.act(input) 2025-12-04T09:46:39.3395263Z 2025-12-04T09:46:39.3395366Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3395725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3396058Z res = mod(**inputs) 2025-12-04T09:46:39.3396421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3396811Z outputs = self.model.decoder( 2025-12-04T09:46:39.3397190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3397567Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3397920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3398274Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3398635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3398996Z return func(*args, **kwargs) 2025-12-04T09:46:39.3399361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3399744Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3399881Z 2025-12-04T09:46:39.3399983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3400328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3400640Z res = mod(**inputs) 2025-12-04T09:46:39.3400985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3401353Z outputs = self.model.decoder( 2025-12-04T09:46:39.3401720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3402094Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3402424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3402786Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3403164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3403555Z return func(*args, **kwargs) 2025-12-04T09:46:39.3403922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3404388Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3404786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3405156Z return func(*args, **kwargs) 2025-12-04T09:46:39.3405520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3405959Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3406133Z 2025-12-04T09:46:39.3406250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3406640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3406989Z res = mod(**inputs) 2025-12-04T09:46:39.3407375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3407797Z outputs = self.model.decoder( 2025-12-04T09:46:39.3408194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3408607Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3408984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3409361Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3409766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3410179Z return func(*args, **kwargs) 2025-12-04T09:46:39.3410576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3411010Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3411506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3411919Z return func(*args, **kwargs) 2025-12-04T09:46:39.3412309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3412742Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3412899Z 2025-12-04T09:46:39.3413023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3413401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3413740Z res = mod(**inputs) 2025-12-04T09:46:39.3414128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3414542Z outputs = self.model.decoder( 2025-12-04T09:46:39.3414946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3415349Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3415718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3416105Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3416500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3416892Z return func(*args, **kwargs) 2025-12-04T09:46:39.3417286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3417726Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3418174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3418568Z return func(*args, **kwargs) 2025-12-04T09:46:39.3418979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3419404Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3419551Z 2025-12-04T09:46:39.3419634Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3419860Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3420079Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3420316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3420689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3421038Z res = mod(**inputs) 2025-12-04T09:46:39.3421431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3421845Z outputs = self.model.decoder( 2025-12-04T09:46:39.3422245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3422652Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3423010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3423392Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3423788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3424186Z return func(*args, **kwargs) 2025-12-04T09:46:39.3424569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3425034Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3425456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3425839Z return func(*args, **kwargs) 2025-12-04T09:46:39.3426230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3426654Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3426799Z 2025-12-04T09:46:39.3426916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3427285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3427622Z res = mod(**inputs) 2025-12-04T09:46:39.3428000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3428403Z outputs = self.model.decoder( 2025-12-04T09:46:39.3428782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3429167Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3429516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3429871Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3430255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3430629Z return func(*args, **kwargs) 2025-12-04T09:46:39.3431002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3431426Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3431602Z 2025-12-04T09:46:39.3431706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3432063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3432393Z res = mod(**inputs) 2025-12-04T09:46:39.3432756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3433169Z outputs = self.model.decoder( 2025-12-04T09:46:39.3433549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3433930Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3434281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3434648Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3435024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3435401Z return func(*args, **kwargs) 2025-12-04T09:46:39.3435796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3436230Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3436611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3436956Z return self.act(input) 2025-12-04T09:46:39.3437064Z 2025-12-04T09:46:39.3437173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3437533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3437847Z res = mod(**inputs) 2025-12-04T09:46:39.3438207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3438622Z outputs = self.model.decoder( 2025-12-04T09:46:39.3438999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3439386Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3439737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3440104Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3440476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3440846Z return func(*args, **kwargs) 2025-12-04T09:46:39.3441219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3441611Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3441784Z 2025-12-04T09:46:39.3441890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3442248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3442574Z res = mod(**inputs) 2025-12-04T09:46:39.3442931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3443327Z outputs = self.model.decoder( 2025-12-04T09:46:39.3443710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3444096Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3444442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3444804Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3445183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3445553Z return func(*args, **kwargs) 2025-12-04T09:46:39.3445936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3446372Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3446777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3447164Z return func(*args, **kwargs) 2025-12-04T09:46:39.3447537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3447959Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3448121Z 2025-12-04T09:46:39.3448233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3448592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3448914Z res = mod(**inputs) 2025-12-04T09:46:39.3449295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3449679Z outputs = self.model.decoder( 2025-12-04T09:46:39.3450075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3450496Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3450858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3451243Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3451741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3452155Z return func(*args, **kwargs) 2025-12-04T09:46:39.3452541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3453026Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3453457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3453861Z return func(*args, **kwargs) 2025-12-04T09:46:39.3454259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3454681Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3454824Z 2025-12-04T09:46:39.3454941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3455311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3455650Z res = mod(**inputs) 2025-12-04T09:46:39.3456035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3456457Z outputs = self.model.decoder( 2025-12-04T09:46:39.3456859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3457276Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3457642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3458026Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3458414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3458809Z return func(*args, **kwargs) 2025-12-04T09:46:39.3459202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3459651Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3460072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3460476Z return func(*args, **kwargs) 2025-12-04T09:46:39.3460914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3461358Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3461538Z 2025-12-04T09:46:39.3461623Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3461852Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3462066Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3462313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3462689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3463036Z res = mod(**inputs) 2025-12-04T09:46:39.3463390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3463782Z outputs = self.model.decoder( 2025-12-04T09:46:39.3464179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3464561Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3464909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3465270Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3465648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3466006Z return func(*args, **kwargs) 2025-12-04T09:46:39.3466374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3466781Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3467192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3467563Z return func(*args, **kwargs) 2025-12-04T09:46:39.3467934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3468327Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3468465Z 2025-12-04T09:46:39.3468566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3468918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3469238Z res = mod(**inputs) 2025-12-04T09:46:39.3469596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3469977Z outputs = self.model.decoder( 2025-12-04T09:46:39.3470357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3470742Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3471085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3471450Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3471824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3472194Z return func(*args, **kwargs) 2025-12-04T09:46:39.3472692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3473132Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3473306Z 2025-12-04T09:46:39.3473420Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3473773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3474097Z res = mod(**inputs) 2025-12-04T09:46:39.3474459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3474903Z outputs = self.model.decoder( 2025-12-04T09:46:39.3475278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3475695Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3476046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3476407Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3476781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3477153Z return func(*args, **kwargs) 2025-12-04T09:46:39.3477528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3477949Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3478363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3478706Z return self.act(input) 2025-12-04T09:46:39.3478815Z 2025-12-04T09:46:39.3478926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3479276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3479610Z res = mod(**inputs) 2025-12-04T09:46:39.3479962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3480333Z outputs = self.model.decoder( 2025-12-04T09:46:39.3480705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3481111Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3481452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3481798Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3482163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3482527Z return func(*args, **kwargs) 2025-12-04T09:46:39.3482890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3483263Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3483402Z 2025-12-04T09:46:39.3483501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3483844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3484149Z res = mod(**inputs) 2025-12-04T09:46:39.3484501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3484878Z outputs = self.model.decoder( 2025-12-04T09:46:39.3485244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3485612Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3485951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3486303Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3486660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3487024Z return func(*args, **kwargs) 2025-12-04T09:46:39.3487389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3487792Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3488173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3488563Z return func(*args, **kwargs) 2025-12-04T09:46:39.3488939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3489378Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3489541Z 2025-12-04T09:46:39.3489643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3489998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3490312Z res = mod(**inputs) 2025-12-04T09:46:39.3490656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3491043Z outputs = self.model.decoder( 2025-12-04T09:46:39.3491499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3491938Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3492315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3492715Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3493126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3493520Z return func(*args, **kwargs) 2025-12-04T09:46:39.3493924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3494371Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3494802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3495218Z return func(*args, **kwargs) 2025-12-04T09:46:39.3495624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3496054Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3496195Z 2025-12-04T09:46:39.3496313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3496696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3497034Z res = mod(**inputs) 2025-12-04T09:46:39.3497409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3497866Z outputs = self.model.decoder( 2025-12-04T09:46:39.3498274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3498693Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3499062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3499439Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3499847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3500302Z return func(*args, **kwargs) 2025-12-04T09:46:39.3500696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3501139Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3501562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3501959Z return func(*args, **kwargs) 2025-12-04T09:46:39.3502355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3502791Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3502941Z 2025-12-04T09:46:39.3503031Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3503273Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3503490Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3503754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3504128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3504459Z res = mod(**inputs) 2025-12-04T09:46:39.3504835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3505240Z outputs = self.model.decoder( 2025-12-04T09:46:39.3505630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3506034Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3506406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3506766Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3507125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3507490Z return func(*args, **kwargs) 2025-12-04T09:46:39.3507852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3508256Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3508635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3508996Z return func(*args, **kwargs) 2025-12-04T09:46:39.3509355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3509762Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3509903Z 2025-12-04T09:46:39.3510006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3510354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3510666Z res = mod(**inputs) 2025-12-04T09:46:39.3511003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3511379Z outputs = self.model.decoder( 2025-12-04T09:46:39.3511745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3512114Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3512453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3512804Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3513168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3513522Z return func(*args, **kwargs) 2025-12-04T09:46:39.3513884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3514305Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3514471Z 2025-12-04T09:46:39.3514578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3514915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3515226Z res = mod(**inputs) 2025-12-04T09:46:39.3515573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3515942Z outputs = self.model.decoder( 2025-12-04T09:46:39.3516311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3516709Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3517048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3517416Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3517791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3518154Z return func(*args, **kwargs) 2025-12-04T09:46:39.3518509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3518922Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3519300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3519628Z return self.act(input) 2025-12-04T09:46:39.3519733Z 2025-12-04T09:46:39.3519852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3520211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3520528Z res = mod(**inputs) 2025-12-04T09:46:39.3520891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3521251Z outputs = self.model.decoder( 2025-12-04T09:46:39.3521618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3521996Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3522328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3522703Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3523074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3523438Z return func(*args, **kwargs) 2025-12-04T09:46:39.3523791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3524174Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3524306Z 2025-12-04T09:46:39.3524413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3524751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3525063Z res = mod(**inputs) 2025-12-04T09:46:39.3525409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3525786Z outputs = self.model.decoder( 2025-12-04T09:46:39.3526037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3526107Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3526331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3526406Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3526650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3526717Z return func(*args, **kwargs) 2025-12-04T09:46:39.3526961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3527064Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3527299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3527367Z return func(*args, **kwargs) 2025-12-04T09:46:39.3527621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3527746Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3527751Z 2025-12-04T09:46:39.3527860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3528073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3528136Z res = mod(**inputs) 2025-12-04T09:46:39.3528391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3528461Z outputs = self.model.decoder( 2025-12-04T09:46:39.3528712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3528783Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3529001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3529103Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3529346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3529418Z return func(*args, **kwargs) 2025-12-04T09:46:39.3529675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3529775Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3530021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3530087Z return func(*args, **kwargs) 2025-12-04T09:46:39.3530336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3530444Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3530448Z 2025-12-04T09:46:39.3530552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3530759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3530825Z res = mod(**inputs) 2025-12-04T09:46:39.3531076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3531156Z outputs = self.model.decoder( 2025-12-04T09:46:39.3531406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3531542Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3531796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3531887Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3532160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3532235Z return func(*args, **kwargs) 2025-12-04T09:46:39.3532522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3532634Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3532887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3532967Z return func(*args, **kwargs) 2025-12-04T09:46:39.3533233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3533318Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3533323Z 2025-12-04T09:46:39.3533414Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3533494Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3533571Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3533684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3533902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3533969Z res = mod(**inputs) 2025-12-04T09:46:39.3534252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3534335Z outputs = self.model.decoder( 2025-12-04T09:46:39.3534581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3534650Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3534858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3534942Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3535214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3535282Z return func(*args, **kwargs) 2025-12-04T09:46:39.3535532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3535626Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3535865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3535929Z return func(*args, **kwargs) 2025-12-04T09:46:39.3536172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3536256Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3536260Z 2025-12-04T09:46:39.3536382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3536575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3536637Z res = mod(**inputs) 2025-12-04T09:46:39.3536878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3536957Z outputs = self.model.decoder( 2025-12-04T09:46:39.3537261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3537331Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3537550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3537623Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3537861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3537927Z return func(*args, **kwargs) 2025-12-04T09:46:39.3538168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3538289Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3538292Z 2025-12-04T09:46:39.3538390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3538593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3538651Z res = mod(**inputs) 2025-12-04T09:46:39.3538891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3538965Z outputs = self.model.decoder( 2025-12-04T09:46:39.3539204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3539272Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3539487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3539559Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3539812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3539898Z return func(*args, **kwargs) 2025-12-04T09:46:39.3540136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3540253Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3540455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3540519Z return self.act(input) 2025-12-04T09:46:39.3540528Z 2025-12-04T09:46:39.3540624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3540810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3540876Z res = mod(**inputs) 2025-12-04T09:46:39.3541139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3541208Z outputs = self.model.decoder( 2025-12-04T09:46:39.3541456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3541524Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3541739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3541814Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3542041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3542131Z return func(*args, **kwargs) 2025-12-04T09:46:39.3542382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3542459Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3542469Z 2025-12-04T09:46:39.3542566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3542755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3542824Z res = mod(**inputs) 2025-12-04T09:46:39.3543067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3543134Z outputs = self.model.decoder( 2025-12-04T09:46:39.3543381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3543448Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3543669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3543745Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3543975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3544047Z return func(*args, **kwargs) 2025-12-04T09:46:39.3544292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3544384Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3544623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3544687Z return func(*args, **kwargs) 2025-12-04T09:46:39.3544934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3545043Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3545047Z 2025-12-04T09:46:39.3545145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3545357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3545422Z res = mod(**inputs) 2025-12-04T09:46:39.3545667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3545762Z outputs = self.model.decoder( 2025-12-04T09:46:39.3546022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3546097Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3546312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3546384Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3546625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3546705Z return func(*args, **kwargs) 2025-12-04T09:46:39.3546950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3547044Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3547276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3547346Z return func(*args, **kwargs) 2025-12-04T09:46:39.3547584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3547658Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3547669Z 2025-12-04T09:46:39.3547767Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3547969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3548037Z res = mod(**inputs) 2025-12-04T09:46:39.3548277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3548344Z outputs = self.model.decoder( 2025-12-04T09:46:39.3548586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3548654Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3548868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3548941Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3549166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3549238Z return func(*args, **kwargs) 2025-12-04T09:46:39.3549476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3549568Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3549801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3549867Z return func(*args, **kwargs) 2025-12-04T09:46:39.3550110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3550190Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3550193Z 2025-12-04T09:46:39.3550266Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3550348Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3550419Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3550514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3550709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3550770Z res = mod(**inputs) 2025-12-04T09:46:39.3551033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3551100Z outputs = self.model.decoder( 2025-12-04T09:46:39.3551359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3551434Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3551642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3551713Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3551949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3552013Z return func(*args, **kwargs) 2025-12-04T09:46:39.3552273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3552368Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3552594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3552663Z return func(*args, **kwargs) 2025-12-04T09:46:39.3552899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3552979Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3552982Z 2025-12-04T09:46:39.3553078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3553263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3553326Z res = mod(**inputs) 2025-12-04T09:46:39.3553592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3553661Z outputs = self.model.decoder( 2025-12-04T09:46:39.3553909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3553975Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3554190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3554261Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3554489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3554557Z return func(*args, **kwargs) 2025-12-04T09:46:39.3554794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3554911Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3554915Z 2025-12-04T09:46:39.3555013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3555196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3555262Z res = mod(**inputs) 2025-12-04T09:46:39.3555497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3555563Z outputs = self.model.decoder( 2025-12-04T09:46:39.3555804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3555869Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3556082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3556153Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3556380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3556452Z return func(*args, **kwargs) 2025-12-04T09:46:39.3556705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3556833Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3557043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3557110Z return self.act(input) 2025-12-04T09:46:39.3557113Z 2025-12-04T09:46:39.3557218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3557405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3557465Z res = mod(**inputs) 2025-12-04T09:46:39.3557717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3557787Z outputs = self.model.decoder( 2025-12-04T09:46:39.3558055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3558126Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3558343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3558427Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3558663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3558729Z return func(*args, **kwargs) 2025-12-04T09:46:39.3559031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3559128Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3559132Z 2025-12-04T09:46:39.3559237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3559430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3559491Z res = mod(**inputs) 2025-12-04T09:46:39.3559741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3559812Z outputs = self.model.decoder( 2025-12-04T09:46:39.3560065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3560132Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3560346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3560428Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3560661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3560726Z return func(*args, **kwargs) 2025-12-04T09:46:39.3560978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3561073Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3561313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3561376Z return func(*args, **kwargs) 2025-12-04T09:46:39.3561622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T09:46:39.3561736Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:46:39.3561740Z 2025-12-04T09:46:39.3561835Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3562030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3562101Z res = mod(**inputs) 2025-12-04T09:46:39.3562371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3562455Z outputs = self.model.decoder( 2025-12-04T09:46:39.3562706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3562809Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3563109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3563217Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3563519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3563598Z return func(*args, **kwargs) 2025-12-04T09:46:39.3564112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3564299Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3564581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3564703Z return func(*args, **kwargs) 2025-12-04T09:46:39.3564972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T09:46:39.3565063Z key_states = self.k_proj(current_states) 2025-12-04T09:46:39.3565067Z 2025-12-04T09:46:39.3565262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3565484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3590541Z res = mod(**inputs) 2025-12-04T09:46:39.3591018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3591299Z outputs = self.model.decoder( 2025-12-04T09:46:39.3591595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3591676Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3591930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3592021Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3592280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3592366Z return func(*args, **kwargs) 2025-12-04T09:46:39.3592625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3592733Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3592995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3593064Z return func(*args, **kwargs) 2025-12-04T09:46:39.3593325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T09:46:39.3593419Z value_states = self.v_proj(current_states) 2025-12-04T09:46:39.3593426Z 2025-12-04T09:46:39.3593512Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3593599Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3593676Z cudagraph partition due to non gpu ops 2025-12-04T09:46:39.3593787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3594006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3594075Z res = mod(**inputs) 2025-12-04T09:46:39.3594347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3594427Z outputs = self.model.decoder( 2025-12-04T09:46:39.3594726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3594810Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3595091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3595174Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3595423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3595493Z return func(*args, **kwargs) 2025-12-04T09:46:39.3595752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T09:46:39.3595855Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:46:39.3596134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3596215Z return func(*args, **kwargs) 2025-12-04T09:46:39.3596470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T09:46:39.3596566Z attn_output = self.out_proj(attn_output) 2025-12-04T09:46:39.3596571Z 2025-12-04T09:46:39.3596679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3596886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3596959Z res = mod(**inputs) 2025-12-04T09:46:39.3597219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3597295Z outputs = self.model.decoder( 2025-12-04T09:46:39.3597573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3597648Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3597879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3597956Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3598197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3598272Z return func(*args, **kwargs) 2025-12-04T09:46:39.3598523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3598647Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3598659Z 2025-12-04T09:46:39.3598765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3598966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3599037Z res = mod(**inputs) 2025-12-04T09:46:39.3599290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3599365Z outputs = self.model.decoder( 2025-12-04T09:46:39.3599622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3599695Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3599925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3600004Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3600245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3600322Z return func(*args, **kwargs) 2025-12-04T09:46:39.3600573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T09:46:39.3600696Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:46:39.3600947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:46:39.3601041Z return self.act(input) 2025-12-04T09:46:39.3601045Z 2025-12-04T09:46:39.3601163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3601383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3601446Z res = mod(**inputs) 2025-12-04T09:46:39.3601709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T09:46:39.3601783Z outputs = self.model.decoder( 2025-12-04T09:46:39.3602045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T09:46:39.3602119Z layer_outputs = decoder_layer( 2025-12-04T09:46:39.3602369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:46:39.3602456Z return super().__call__(*args, **kwargs) 2025-12-04T09:46:39.3602718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:46:39.3602794Z return func(*args, **kwargs) 2025-12-04T09:46:39.3603069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T09:46:39.3603157Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:46:39.3603161Z 2025-12-04T09:46:39.3603284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3603482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3603583Z res = mod(**inputs) 2025-12-04T09:46:39.3603854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 844, in forward 2025-12-04T09:46:39.3603947Z logits = self.output_projection(outputs[0]) 2025-12-04T09:46:39.3603951Z 2025-12-04T09:46:39.3604060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:46:39.3604259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:46:39.3604322Z res = mod(**inputs) 2025-12-04T09:46:39.3604601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 849, in forward 2025-12-04T09:46:39.3604756Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:46:39.3604761Z 2025-12-04T09:46:49.5619370Z Compilation time (from dynamo_timed): 16.856881133 2025-12-04T09:46:49.5656446Z pass 2025-12-04T09:46:49.5656942Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:46:49.5657941Z TIMING: _recursive_pre_grad_passes:0.00771 _recursive_joint_graph_passes:0.73767 _recursive_post_grad_passes:0.0664 async_compile.wait:0.8536 code_gen:9.42473 inductor_compile:10.67417 backend_compile:13.97962 gc:0.00013 entire_frame_compile:16.85688 total_wall_time:16.85688 2025-12-04T09:46:49.5659016Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:8334 | FakeTensor.__torch_dispatch__:4316 | ProxyTorchDispatchMode.__torch_dispatch__:2529 2025-12-04T09:46:49.5659564Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-12-04T09:46:52.0186323Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:46:52.0187963Z import pynvml # type: ignore[import] 2025-12-04T09:46:55.4136934Z 2025-12-04T09:47:00.7376390Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:47:00.7376736Z loading model: 0it [00:05, ?it/s] 2025-12-04T09:47:00.7401125Z cpu eval XGLMForCausalLM 2025-12-04T09:47:01.1267422Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-12-04T09:47:01.2265597Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:01.7751399Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:02.2984539Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:16.8309751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8310365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8310782Z res = mod(**inputs) 2025-12-04T09:47:16.8311648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8312081Z outputs = self.model( 2025-12-04T09:47:16.8312461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8312898Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8313278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8313646Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8314044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8314422Z return func(*args, **kwargs) 2025-12-04T09:47:16.8314796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8315328Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8315737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8316112Z return func(*args, **kwargs) 2025-12-04T09:47:16.8316520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8316959Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8317135Z 2025-12-04T09:47:16.8317244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8317637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8318076Z res = mod(**inputs) 2025-12-04T09:47:16.8318463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8318871Z outputs = self.model( 2025-12-04T09:47:16.8319254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8319658Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8320037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8320410Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8320786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8321162Z return func(*args, **kwargs) 2025-12-04T09:47:16.8321534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8321996Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8322422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8322830Z return func(*args, **kwargs) 2025-12-04T09:47:16.8323279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8323699Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8323897Z 2025-12-04T09:47:16.8324010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8324396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8324740Z res = mod(**inputs) 2025-12-04T09:47:16.8325111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8325511Z outputs = self.model( 2025-12-04T09:47:16.8325894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8326298Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8326683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8327079Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8327476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8327863Z return func(*args, **kwargs) 2025-12-04T09:47:16.8328243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8328678Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8329099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8329480Z return func(*args, **kwargs) 2025-12-04T09:47:16.8329865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8330339Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8330514Z 2025-12-04T09:47:16.8330635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8331006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8331354Z res = mod(**inputs) 2025-12-04T09:47:16.8331888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8332294Z outputs = self.model( 2025-12-04T09:47:16.8332699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8333126Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8333524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8333908Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8334322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8334719Z return func(*args, **kwargs) 2025-12-04T09:47:16.8335112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8335541Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8335968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8336366Z return func(*args, **kwargs) 2025-12-04T09:47:16.8336753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8337227Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8337437Z 2025-12-04T09:47:16.8337548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8337930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8338314Z res = mod(**inputs) 2025-12-04T09:47:16.8338697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8339117Z outputs = self.model( 2025-12-04T09:47:16.8339488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8339892Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8340258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8340641Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8341031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8341424Z return func(*args, **kwargs) 2025-12-04T09:47:16.8341836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8342272Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8342693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8343087Z return func(*args, **kwargs) 2025-12-04T09:47:16.8343493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8343902Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8344054Z 2025-12-04T09:47:16.8344163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8344521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8344862Z res = mod(**inputs) 2025-12-04T09:47:16.8345209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8345594Z outputs = self.model( 2025-12-04T09:47:16.8345957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8346333Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8346689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8347075Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8347471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8347904Z return func(*args, **kwargs) 2025-12-04T09:47:16.8348296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8348725Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8349124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8349487Z return func(*args, **kwargs) 2025-12-04T09:47:16.8349861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8350271Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8350420Z 2025-12-04T09:47:16.8350521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8350884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8351207Z res = mod(**inputs) 2025-12-04T09:47:16.8351561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8351955Z outputs = self.model( 2025-12-04T09:47:16.8352348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8352748Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8353093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8353481Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8353875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8354267Z return func(*args, **kwargs) 2025-12-04T09:47:16.8354655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8355089Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8355513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8355879Z return func(*args, **kwargs) 2025-12-04T09:47:16.8356256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8356698Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8356890Z 2025-12-04T09:47:16.8357007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8357379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8357716Z res = mod(**inputs) 2025-12-04T09:47:16.8358088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8358485Z outputs = self.model( 2025-12-04T09:47:16.8358869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8359296Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8359664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8360046Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8360436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8360831Z return func(*args, **kwargs) 2025-12-04T09:47:16.8361214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8361632Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8362047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8362435Z return func(*args, **kwargs) 2025-12-04T09:47:16.8362822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8363225Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8363375Z 2025-12-04T09:47:16.8363483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8363856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8364183Z res = mod(**inputs) 2025-12-04T09:47:16.8364551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8364941Z outputs = self.model( 2025-12-04T09:47:16.8365313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8365704Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8366077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8366463Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8366900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8367291Z return func(*args, **kwargs) 2025-12-04T09:47:16.8367675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8368147Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8368330Z 2025-12-04T09:47:16.8368440Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8368822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8369159Z res = mod(**inputs) 2025-12-04T09:47:16.8369532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8369943Z outputs = self.model( 2025-12-04T09:47:16.8370355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8370773Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8371148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8371644Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8372062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8372638Z return func(*args, **kwargs) 2025-12-04T09:47:16.8373035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8373507Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8373987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8374369Z return self.act(input) 2025-12-04T09:47:16.8374486Z 2025-12-04T09:47:16.8374599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8374988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8375349Z res = mod(**inputs) 2025-12-04T09:47:16.8375721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8376120Z outputs = self.model( 2025-12-04T09:47:16.8376498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8376899Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8377261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8377648Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8378059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8378467Z return func(*args, **kwargs) 2025-12-04T09:47:16.8378859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8379279Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8379423Z 2025-12-04T09:47:16.8379536Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8379906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8380242Z res = mod(**inputs) 2025-12-04T09:47:16.8380612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8381005Z outputs = self.model( 2025-12-04T09:47:16.8381389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8381814Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8382235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8382618Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8383067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8383537Z return func(*args, **kwargs) 2025-12-04T09:47:16.8383923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8384375Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8384810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8385216Z return func(*args, **kwargs) 2025-12-04T09:47:16.8385636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8386113Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8386303Z 2025-12-04T09:47:16.8386415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8386809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8387171Z res = mod(**inputs) 2025-12-04T09:47:16.8387557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8387966Z outputs = self.model( 2025-12-04T09:47:16.8388355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8388768Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8389170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8389585Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8389992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8390400Z return func(*args, **kwargs) 2025-12-04T09:47:16.8390801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8391254Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8391682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8392094Z return func(*args, **kwargs) 2025-12-04T09:47:16.8392491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8392919Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8393604Z 2025-12-04T09:47:16.8393720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8394120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8394477Z res = mod(**inputs) 2025-12-04T09:47:16.8394860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8395267Z outputs = self.model( 2025-12-04T09:47:16.8395657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8396069Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8396440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8396835Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8397264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8397667Z return func(*args, **kwargs) 2025-12-04T09:47:16.8398091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8398553Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8398986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8399383Z return func(*args, **kwargs) 2025-12-04T09:47:16.8399790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8400253Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8400429Z 2025-12-04T09:47:16.8400547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8400933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8402144Z res = mod(**inputs) 2025-12-04T09:47:16.8402549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8402945Z outputs = self.model( 2025-12-04T09:47:16.8403332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8403736Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8404112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8404503Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8404919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8405359Z return func(*args, **kwargs) 2025-12-04T09:47:16.8405758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8406191Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8406620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8407024Z return func(*args, **kwargs) 2025-12-04T09:47:16.8407425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8407913Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8408122Z 2025-12-04T09:47:16.8408235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8408630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8408992Z res = mod(**inputs) 2025-12-04T09:47:16.8409373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8409798Z outputs = self.model( 2025-12-04T09:47:16.8410176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8410591Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8410970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8411363Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8411870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8412289Z return func(*args, **kwargs) 2025-12-04T09:47:16.8412698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8413164Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8413589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8414027Z return func(*args, **kwargs) 2025-12-04T09:47:16.8414432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8414886Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8415054Z 2025-12-04T09:47:16.8415166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8415538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8415881Z res = mod(**inputs) 2025-12-04T09:47:16.8416244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8416643Z outputs = self.model( 2025-12-04T09:47:16.8417020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8417440Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8417806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8418187Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8418589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8418970Z return func(*args, **kwargs) 2025-12-04T09:47:16.8419361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8419788Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8420210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8420622Z return func(*args, **kwargs) 2025-12-04T09:47:16.8421011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8421440Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8421595Z 2025-12-04T09:47:16.8421702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8422080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8422430Z res = mod(**inputs) 2025-12-04T09:47:16.8422799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8423185Z outputs = self.model( 2025-12-04T09:47:16.8423559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8423959Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8424316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8424698Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8425099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8425490Z return func(*args, **kwargs) 2025-12-04T09:47:16.8425868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8426297Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8426715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8427101Z return func(*args, **kwargs) 2025-12-04T09:47:16.8427478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8427938Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8428126Z 2025-12-04T09:47:16.8428244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8428631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8428987Z res = mod(**inputs) 2025-12-04T09:47:16.8429358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8429757Z outputs = self.model( 2025-12-04T09:47:16.8430131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8430530Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8430903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8431286Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8431692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8432086Z return func(*args, **kwargs) 2025-12-04T09:47:16.8432472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8432895Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8433312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8433698Z return func(*args, **kwargs) 2025-12-04T09:47:16.8434083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8434485Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8434634Z 2025-12-04T09:47:16.8434762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8435145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8435475Z res = mod(**inputs) 2025-12-04T09:47:16.8435854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8436255Z outputs = self.model( 2025-12-04T09:47:16.8436634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8437027Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8437394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8437780Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8438170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8438567Z return func(*args, **kwargs) 2025-12-04T09:47:16.8438958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8439411Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8439592Z 2025-12-04T09:47:16.8439699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8440078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8440424Z res = mod(**inputs) 2025-12-04T09:47:16.8440796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8441198Z outputs = self.model( 2025-12-04T09:47:16.8441596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8442000Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8442366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8442753Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8443172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8443568Z return func(*args, **kwargs) 2025-12-04T09:47:16.8443972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8444422Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8444830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8445184Z return self.act(input) 2025-12-04T09:47:16.8445306Z 2025-12-04T09:47:16.8445414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8445792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8446128Z res = mod(**inputs) 2025-12-04T09:47:16.8446512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8446916Z outputs = self.model( 2025-12-04T09:47:16.8447297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8447702Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8448064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8448449Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8448848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8449239Z return func(*args, **kwargs) 2025-12-04T09:47:16.8449656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8450070Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8450219Z 2025-12-04T09:47:16.8450338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8450714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8451054Z res = mod(**inputs) 2025-12-04T09:47:16.8451513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8451928Z outputs = self.model( 2025-12-04T09:47:16.8452315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8452739Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8453104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8453488Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8453907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8454314Z return func(*args, **kwargs) 2025-12-04T09:47:16.8454704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8455148Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8455584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8455991Z return func(*args, **kwargs) 2025-12-04T09:47:16.8456384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8456840Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8457025Z 2025-12-04T09:47:16.8457137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8457525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8457887Z res = mod(**inputs) 2025-12-04T09:47:16.8458280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8458709Z outputs = self.model( 2025-12-04T09:47:16.8459087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8459500Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8459879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8460269Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8460669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8461074Z return func(*args, **kwargs) 2025-12-04T09:47:16.8461493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8461928Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8462365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8462762Z return func(*args, **kwargs) 2025-12-04T09:47:16.8463162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8463573Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8463727Z 2025-12-04T09:47:16.8463838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8464226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8464593Z res = mod(**inputs) 2025-12-04T09:47:16.8464969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8465375Z outputs = self.model( 2025-12-04T09:47:16.8465759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8466163Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8466529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8466904Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8467296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8467676Z return func(*args, **kwargs) 2025-12-04T09:47:16.8468063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8468487Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8468898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8469289Z return func(*args, **kwargs) 2025-12-04T09:47:16.8469675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8470111Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8470280Z 2025-12-04T09:47:16.8470387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8470761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8471094Z res = mod(**inputs) 2025-12-04T09:47:16.8471465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8471852Z outputs = self.model( 2025-12-04T09:47:16.8472435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8472864Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8473229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8473676Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8474077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8474468Z return func(*args, **kwargs) 2025-12-04T09:47:16.8474846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8475271Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8475693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8476120Z return func(*args, **kwargs) 2025-12-04T09:47:16.8476518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8476990Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8477190Z 2025-12-04T09:47:16.8477306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8477672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8478015Z res = mod(**inputs) 2025-12-04T09:47:16.8478384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8478777Z outputs = self.model( 2025-12-04T09:47:16.8479145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8479580Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8479974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8480351Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8480759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8481158Z return func(*args, **kwargs) 2025-12-04T09:47:16.8481553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8481976Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8482403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8482801Z return func(*args, **kwargs) 2025-12-04T09:47:16.8483208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8483650Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8483810Z 2025-12-04T09:47:16.8483922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8484308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8484663Z res = mod(**inputs) 2025-12-04T09:47:16.8485060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8485478Z outputs = self.model( 2025-12-04T09:47:16.8485865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8486282Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8486659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8487056Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8487498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8487912Z return func(*args, **kwargs) 2025-12-04T09:47:16.8488314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8488779Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8489205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8489608Z return func(*args, **kwargs) 2025-12-04T09:47:16.8490009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8490456Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8490630Z 2025-12-04T09:47:16.8490743Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8491151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8491585Z res = mod(**inputs) 2025-12-04T09:47:16.8491993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8492432Z outputs = self.model( 2025-12-04T09:47:16.8492845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8493357Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8493751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8494173Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8494611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8495034Z return func(*args, **kwargs) 2025-12-04T09:47:16.8495448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8495907Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8496348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8496747Z return func(*args, **kwargs) 2025-12-04T09:47:16.8497153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8497628Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8497822Z 2025-12-04T09:47:16.8497943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8498333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8498686Z res = mod(**inputs) 2025-12-04T09:47:16.8499083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8499490Z outputs = self.model( 2025-12-04T09:47:16.8499890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8500311Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8500684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8501069Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8501474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8501870Z return func(*args, **kwargs) 2025-12-04T09:47:16.8502259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8502693Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8503135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8503523Z return func(*args, **kwargs) 2025-12-04T09:47:16.8503922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8504331Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8504473Z 2025-12-04T09:47:16.8504588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8504960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8505290Z res = mod(**inputs) 2025-12-04T09:47:16.8505657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8506055Z outputs = self.model( 2025-12-04T09:47:16.8506441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8506845Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8507210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8507590Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8507978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8508368Z return func(*args, **kwargs) 2025-12-04T09:47:16.8508754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8509197Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8509411Z 2025-12-04T09:47:16.8509519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8509897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8510234Z res = mod(**inputs) 2025-12-04T09:47:16.8510600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8511000Z outputs = self.model( 2025-12-04T09:47:16.8511378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8511774Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8512150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8512531Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8512928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8513318Z return func(*args, **kwargs) 2025-12-04T09:47:16.8513677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8514093Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8514477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8514810Z return self.act(input) 2025-12-04T09:47:16.8514928Z 2025-12-04T09:47:16.8515031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8515385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8515695Z res = mod(**inputs) 2025-12-04T09:47:16.8516045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8516428Z outputs = self.model( 2025-12-04T09:47:16.8516774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8517159Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8517501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8517919Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8518283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8518643Z return func(*args, **kwargs) 2025-12-04T09:47:16.8519002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8519383Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8519517Z 2025-12-04T09:47:16.8519618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8519968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8520292Z res = mod(**inputs) 2025-12-04T09:47:16.8520633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8521003Z outputs = self.model( 2025-12-04T09:47:16.8521354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8521728Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8522059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8522408Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8522776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8523168Z return func(*args, **kwargs) 2025-12-04T09:47:16.8523588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.8523971Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.8524101Z 2025-12-04T09:47:16.8524208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8524556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8524877Z res = mod(**inputs) 2025-12-04T09:47:16.8525226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8525604Z outputs = self.model( 2025-12-04T09:47:16.8525958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8526339Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8526689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8527047Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8527428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8527799Z return func(*args, **kwargs) 2025-12-04T09:47:16.8528169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8528576Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8528976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8529346Z return func(*args, **kwargs) 2025-12-04T09:47:16.8529721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8530161Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8530338Z 2025-12-04T09:47:16.8530447Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8530878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8531215Z res = mod(**inputs) 2025-12-04T09:47:16.8531698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8532162Z outputs = self.model( 2025-12-04T09:47:16.8532556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8532986Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8533359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8533742Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8534141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8534550Z return func(*args, **kwargs) 2025-12-04T09:47:16.8534919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8535322Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8535711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8536081Z return func(*args, **kwargs) 2025-12-04T09:47:16.8536451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8536838Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8536971Z 2025-12-04T09:47:16.8537073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8537456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8537785Z res = mod(**inputs) 2025-12-04T09:47:16.8538183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8538571Z outputs = self.model( 2025-12-04T09:47:16.8538934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8539324Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8539673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8540041Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8540429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8540800Z return func(*args, **kwargs) 2025-12-04T09:47:16.8541181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8541595Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8541998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8542371Z return func(*args, **kwargs) 2025-12-04T09:47:16.8542744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8543172Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8543336Z 2025-12-04T09:47:16.8543449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8543803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8544127Z res = mod(**inputs) 2025-12-04T09:47:16.8544488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8544868Z outputs = self.model( 2025-12-04T09:47:16.8545260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8545650Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8546022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8546380Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8546766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8547151Z return func(*args, **kwargs) 2025-12-04T09:47:16.8547524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8547946Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8548376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8548750Z return func(*args, **kwargs) 2025-12-04T09:47:16.8549108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8549567Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8549761Z 2025-12-04T09:47:16.8549879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8550245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8550555Z res = mod(**inputs) 2025-12-04T09:47:16.8550904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8551277Z outputs = self.model( 2025-12-04T09:47:16.8551652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8552044Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8552405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8552776Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8553154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8553534Z return func(*args, **kwargs) 2025-12-04T09:47:16.8553905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8554310Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8554713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8555090Z return func(*args, **kwargs) 2025-12-04T09:47:16.8555467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8555849Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8555994Z 2025-12-04T09:47:16.8556096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8556448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8556762Z res = mod(**inputs) 2025-12-04T09:47:16.8557101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8557478Z outputs = self.model( 2025-12-04T09:47:16.8557844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8558213Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8558563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8558920Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8559311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8559671Z return func(*args, **kwargs) 2025-12-04T09:47:16.8560047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8560440Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8560816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8561174Z return func(*args, **kwargs) 2025-12-04T09:47:16.8561531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8561926Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8562071Z 2025-12-04T09:47:16.8562187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8562540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8562851Z res = mod(**inputs) 2025-12-04T09:47:16.8563201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8563567Z outputs = self.model( 2025-12-04T09:47:16.8563930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8564306Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8564638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8564992Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8565396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8565774Z return func(*args, **kwargs) 2025-12-04T09:47:16.8566135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8566533Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8566928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8567290Z return func(*args, **kwargs) 2025-12-04T09:47:16.8567654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8568083Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8568260Z 2025-12-04T09:47:16.8568371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8568717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8569038Z res = mod(**inputs) 2025-12-04T09:47:16.8569393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8569770Z outputs = self.model( 2025-12-04T09:47:16.8570120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8570499Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8570855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8571223Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8571736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8572162Z return func(*args, **kwargs) 2025-12-04T09:47:16.8572800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8573290Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8573721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8574148Z return func(*args, **kwargs) 2025-12-04T09:47:16.8574531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8574944Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8575095Z 2025-12-04T09:47:16.8575205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8575579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8575909Z res = mod(**inputs) 2025-12-04T09:47:16.8576262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8576658Z outputs = self.model( 2025-12-04T09:47:16.8577026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8577409Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8577770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8578138Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8578512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8578891Z return func(*args, **kwargs) 2025-12-04T09:47:16.8579272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8579730Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8579899Z 2025-12-04T09:47:16.8580004Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8580362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8580683Z res = mod(**inputs) 2025-12-04T09:47:16.8581030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8581410Z outputs = self.model( 2025-12-04T09:47:16.8581769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8582147Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8582485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8582845Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8583225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8583593Z return func(*args, **kwargs) 2025-12-04T09:47:16.8583955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8584428Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8584840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8585189Z return self.act(input) 2025-12-04T09:47:16.8585307Z 2025-12-04T09:47:16.8585408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8585764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8586088Z res = mod(**inputs) 2025-12-04T09:47:16.8586432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8586810Z outputs = self.model( 2025-12-04T09:47:16.8587202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8587587Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8587934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8588314Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8588690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8589050Z return func(*args, **kwargs) 2025-12-04T09:47:16.8589414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8589797Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8589934Z 2025-12-04T09:47:16.8590035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8590406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8590731Z res = mod(**inputs) 2025-12-04T09:47:16.8591086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8591459Z outputs = self.model( 2025-12-04T09:47:16.8591813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8592193Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8592538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8592895Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8593276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8593667Z return func(*args, **kwargs) 2025-12-04T09:47:16.8594026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8594431Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8594831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8595209Z return func(*args, **kwargs) 2025-12-04T09:47:16.8595566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8595988Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8596146Z 2025-12-04T09:47:16.8596255Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8596590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8596898Z res = mod(**inputs) 2025-12-04T09:47:16.8597241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8597606Z outputs = self.model( 2025-12-04T09:47:16.8597944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8598316Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8598651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8599003Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8599361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8599717Z return func(*args, **kwargs) 2025-12-04T09:47:16.8600072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8600463Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8600882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8601259Z return func(*args, **kwargs) 2025-12-04T09:47:16.8601626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8602029Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8602172Z 2025-12-04T09:47:16.8602274Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8602635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8602953Z res = mod(**inputs) 2025-12-04T09:47:16.8603297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8603664Z outputs = self.model( 2025-12-04T09:47:16.8604030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8604398Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8604738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8605095Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8605452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8605811Z return func(*args, **kwargs) 2025-12-04T09:47:16.8606170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8606562Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8606946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8607326Z return func(*args, **kwargs) 2025-12-04T09:47:16.8607699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8608113Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8608273Z 2025-12-04T09:47:16.8608374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8608727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8609047Z res = mod(**inputs) 2025-12-04T09:47:16.8609388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8609764Z outputs = self.model( 2025-12-04T09:47:16.8610127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8610517Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8610866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8611240Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8611688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8612076Z return func(*args, **kwargs) 2025-12-04T09:47:16.8612439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8612873Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8613297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8613679Z return func(*args, **kwargs) 2025-12-04T09:47:16.8614070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8614538Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8614734Z 2025-12-04T09:47:16.8614874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8615254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8615596Z res = mod(**inputs) 2025-12-04T09:47:16.8615948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8616315Z outputs = self.model( 2025-12-04T09:47:16.8616678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8617060Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8617404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8617757Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8618153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8618524Z return func(*args, **kwargs) 2025-12-04T09:47:16.8618890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8619287Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8619685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8620056Z return func(*args, **kwargs) 2025-12-04T09:47:16.8620416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8620812Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8620981Z 2025-12-04T09:47:16.8621086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8621448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8621768Z res = mod(**inputs) 2025-12-04T09:47:16.8622121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8622501Z outputs = self.model( 2025-12-04T09:47:16.8622853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8623238Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8623587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8623948Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8624321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8624697Z return func(*args, **kwargs) 2025-12-04T09:47:16.8625068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8625476Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8625871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8626238Z return func(*args, **kwargs) 2025-12-04T09:47:16.8626612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8626999Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8627151Z 2025-12-04T09:47:16.8627248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8627595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8627907Z res = mod(**inputs) 2025-12-04T09:47:16.8628247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8628626Z outputs = self.model( 2025-12-04T09:47:16.8628978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8629373Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8629728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8630082Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8630451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8630807Z return func(*args, **kwargs) 2025-12-04T09:47:16.8631166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8631562Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8631964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8632318Z return func(*args, **kwargs) 2025-12-04T09:47:16.8632688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8633111Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8633287Z 2025-12-04T09:47:16.8633391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8633747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8634063Z res = mod(**inputs) 2025-12-04T09:47:16.8634419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8634817Z outputs = self.model( 2025-12-04T09:47:16.8635181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8635551Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8635879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8636233Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8636597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8636954Z return func(*args, **kwargs) 2025-12-04T09:47:16.8637304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8637693Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8638075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8638430Z return func(*args, **kwargs) 2025-12-04T09:47:16.8638779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8639157Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8639290Z 2025-12-04T09:47:16.8639400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8639744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8640067Z res = mod(**inputs) 2025-12-04T09:47:16.8640408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8640778Z outputs = self.model( 2025-12-04T09:47:16.8641122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8641499Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8641861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8642220Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8642597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8642998Z return func(*args, **kwargs) 2025-12-04T09:47:16.8643411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8643832Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8644011Z 2025-12-04T09:47:16.8644112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8644466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8644784Z res = mod(**inputs) 2025-12-04T09:47:16.8645148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8645525Z outputs = self.model( 2025-12-04T09:47:16.8645884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8646261Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8646613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8646983Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8647360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8647726Z return func(*args, **kwargs) 2025-12-04T09:47:16.8648095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8648539Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8648923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8649270Z return self.act(input) 2025-12-04T09:47:16.8649387Z 2025-12-04T09:47:16.8649492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8649850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8650159Z res = mod(**inputs) 2025-12-04T09:47:16.8650512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8650886Z outputs = self.model( 2025-12-04T09:47:16.8651241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8651734Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8652120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8652524Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8652919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8653314Z return func(*args, **kwargs) 2025-12-04T09:47:16.8653711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8654104Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8654242Z 2025-12-04T09:47:16.8654347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8654704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8655028Z res = mod(**inputs) 2025-12-04T09:47:16.8655374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8655753Z outputs = self.model( 2025-12-04T09:47:16.8656134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8656513Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8656876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8657245Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8657626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8657995Z return func(*args, **kwargs) 2025-12-04T09:47:16.8658365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.8658753Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.8658888Z 2025-12-04T09:47:16.8658995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8659361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8659683Z res = mod(**inputs) 2025-12-04T09:47:16.8660035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8660404Z outputs = self.model( 2025-12-04T09:47:16.8660758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8661137Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8661486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8661836Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8662244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8662607Z return func(*args, **kwargs) 2025-12-04T09:47:16.8662968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8663359Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8663755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8664120Z return func(*args, **kwargs) 2025-12-04T09:47:16.8664474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8664897Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8665060Z 2025-12-04T09:47:16.8665161Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8665512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8665817Z res = mod(**inputs) 2025-12-04T09:47:16.8666164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8666533Z outputs = self.model( 2025-12-04T09:47:16.8666875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8667248Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8667591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8667944Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8668306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8668665Z return func(*args, **kwargs) 2025-12-04T09:47:16.8669025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8669418Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8669819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8670183Z return func(*args, **kwargs) 2025-12-04T09:47:16.8670558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8670928Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8671063Z 2025-12-04T09:47:16.8671163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8671507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8671816Z res = mod(**inputs) 2025-12-04T09:47:16.8672155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8672690Z outputs = self.model( 2025-12-04T09:47:16.8673088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8673457Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8673796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8674152Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8674519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8674871Z return func(*args, **kwargs) 2025-12-04T09:47:16.8675232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8675625Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8676035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8676382Z return func(*args, **kwargs) 2025-12-04T09:47:16.8676727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8677124Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8677275Z 2025-12-04T09:47:16.8677373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8677709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8678011Z res = mod(**inputs) 2025-12-04T09:47:16.8678344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8678693Z outputs = self.model( 2025-12-04T09:47:16.8679028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8679389Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8679714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8680059Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8680420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8680772Z return func(*args, **kwargs) 2025-12-04T09:47:16.8681115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8681502Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8681880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8682237Z return func(*args, **kwargs) 2025-12-04T09:47:16.8682580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8683027Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8683209Z 2025-12-04T09:47:16.8683313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8683675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8683990Z res = mod(**inputs) 2025-12-04T09:47:16.8684333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8684697Z outputs = self.model( 2025-12-04T09:47:16.8685035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8685402Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8685743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8686110Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8686479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8686833Z return func(*args, **kwargs) 2025-12-04T09:47:16.8687188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8687569Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8687950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8688301Z return func(*args, **kwargs) 2025-12-04T09:47:16.8688649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8689035Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8689179Z 2025-12-04T09:47:16.8689281Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8689629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8689933Z res = mod(**inputs) 2025-12-04T09:47:16.8690274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8690641Z outputs = self.model( 2025-12-04T09:47:16.8690987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8691351Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8691752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8692125Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8692514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8692902Z return func(*args, **kwargs) 2025-12-04T09:47:16.8693288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8693719Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8694122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8694491Z return func(*args, **kwargs) 2025-12-04T09:47:16.8694860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8695254Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8695397Z 2025-12-04T09:47:16.8695498Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8695841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8696148Z res = mod(**inputs) 2025-12-04T09:47:16.8696501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8696870Z outputs = self.model( 2025-12-04T09:47:16.8697218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8697606Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8697946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8698363Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8698767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8699138Z return func(*args, **kwargs) 2025-12-04T09:47:16.8699496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8699619Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8699857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8699925Z return func(*args, **kwargs) 2025-12-04T09:47:16.8700170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8700290Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8700293Z 2025-12-04T09:47:16.8700401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8700590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8700652Z res = mod(**inputs) 2025-12-04T09:47:16.8700921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8700987Z outputs = self.model( 2025-12-04T09:47:16.8701229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8701319Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8701543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8701628Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8701870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8701937Z return func(*args, **kwargs) 2025-12-04T09:47:16.8702190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8702287Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8702525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8702611Z return func(*args, **kwargs) 2025-12-04T09:47:16.8702850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8702937Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8702941Z 2025-12-04T09:47:16.8703041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8703229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8703297Z res = mod(**inputs) 2025-12-04T09:47:16.8703539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8703609Z outputs = self.model( 2025-12-04T09:47:16.8703852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8703921Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8704162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8704238Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8704489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8704564Z return func(*args, **kwargs) 2025-12-04T09:47:16.8704801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8704922Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8704925Z 2025-12-04T09:47:16.8705024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8705215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8705283Z res = mod(**inputs) 2025-12-04T09:47:16.8705543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8705609Z outputs = self.model( 2025-12-04T09:47:16.8705855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8705926Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8706146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8706219Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8706450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8706522Z return func(*args, **kwargs) 2025-12-04T09:47:16.8706783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8706907Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8707113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8707179Z return self.act(input) 2025-12-04T09:47:16.8707185Z 2025-12-04T09:47:16.8707294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8707484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8707543Z res = mod(**inputs) 2025-12-04T09:47:16.8707789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8707852Z outputs = self.model( 2025-12-04T09:47:16.8708098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8708168Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8708389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8708469Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8708700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8708766Z return func(*args, **kwargs) 2025-12-04T09:47:16.8709005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8709081Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8709085Z 2025-12-04T09:47:16.8709189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8709371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8709433Z res = mod(**inputs) 2025-12-04T09:47:16.8709677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8709740Z outputs = self.model( 2025-12-04T09:47:16.8709997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8710081Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8710295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8710373Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8710607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8710671Z return func(*args, **kwargs) 2025-12-04T09:47:16.8710921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8711015Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8711271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8711338Z return func(*args, **kwargs) 2025-12-04T09:47:16.8711572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8711687Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8711690Z 2025-12-04T09:47:16.8711787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8711976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8712034Z res = mod(**inputs) 2025-12-04T09:47:16.8712267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8712355Z outputs = self.model( 2025-12-04T09:47:16.8712590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8712660Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8712876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8712950Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8713182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8713246Z return func(*args, **kwargs) 2025-12-04T09:47:16.8713477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8713575Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8713799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8713865Z return func(*args, **kwargs) 2025-12-04T09:47:16.8714110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8714187Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8714190Z 2025-12-04T09:47:16.8714294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8714478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8714538Z res = mod(**inputs) 2025-12-04T09:47:16.8714776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8714841Z outputs = self.model( 2025-12-04T09:47:16.8715088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8715161Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8715376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8715477Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8715711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8715805Z return func(*args, **kwargs) 2025-12-04T09:47:16.8716049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8716144Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8716382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8716447Z return func(*args, **kwargs) 2025-12-04T09:47:16.8716682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8716797Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8716819Z 2025-12-04T09:47:16.8716922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8717122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8717185Z res = mod(**inputs) 2025-12-04T09:47:16.8717427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8717499Z outputs = self.model( 2025-12-04T09:47:16.8717739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8717811Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8718035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8718149Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8718396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8718464Z return func(*args, **kwargs) 2025-12-04T09:47:16.8718705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8718806Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8719046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8719112Z return func(*args, **kwargs) 2025-12-04T09:47:16.8719362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8719492Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8719497Z 2025-12-04T09:47:16.8719604Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8719794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8719858Z res = mod(**inputs) 2025-12-04T09:47:16.8720106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8720173Z outputs = self.model( 2025-12-04T09:47:16.8720423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8720493Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8720710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8720789Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8721021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8721090Z return func(*args, **kwargs) 2025-12-04T09:47:16.8721356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8721452Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8721690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8721773Z return func(*args, **kwargs) 2025-12-04T09:47:16.8722018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8722108Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8722112Z 2025-12-04T09:47:16.8722209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8722407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8722472Z res = mod(**inputs) 2025-12-04T09:47:16.8722729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8722805Z outputs = self.model( 2025-12-04T09:47:16.8723042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8723112Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8723333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8723406Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8723643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8723708Z return func(*args, **kwargs) 2025-12-04T09:47:16.8723948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8724074Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8724310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8724375Z return func(*args, **kwargs) 2025-12-04T09:47:16.8724622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8724714Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8724718Z 2025-12-04T09:47:16.8724823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8725013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8725075Z res = mod(**inputs) 2025-12-04T09:47:16.8725325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8725390Z outputs = self.model( 2025-12-04T09:47:16.8725639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8725710Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8725927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8726014Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8726258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8726326Z return func(*args, **kwargs) 2025-12-04T09:47:16.8726587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8726680Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8726930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8727000Z return func(*args, **kwargs) 2025-12-04T09:47:16.8727267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8727403Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8727423Z 2025-12-04T09:47:16.8727525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8727726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8727789Z res = mod(**inputs) 2025-12-04T09:47:16.8728035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8728110Z outputs = self.model( 2025-12-04T09:47:16.8728356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8728429Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8728679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8728760Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8729004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8729073Z return func(*args, **kwargs) 2025-12-04T09:47:16.8729325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8729426Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8729667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8729733Z return func(*args, **kwargs) 2025-12-04T09:47:16.8730004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8730085Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8730089Z 2025-12-04T09:47:16.8730197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8730391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8730455Z res = mod(**inputs) 2025-12-04T09:47:16.8730707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8730773Z outputs = self.model( 2025-12-04T09:47:16.8731023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8731094Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8731312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8731399Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8731724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8731798Z return func(*args, **kwargs) 2025-12-04T09:47:16.8732051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8732171Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8732175Z 2025-12-04T09:47:16.8732282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8732488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8732556Z res = mod(**inputs) 2025-12-04T09:47:16.8732834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8732907Z outputs = self.model( 2025-12-04T09:47:16.8733190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8733284Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8733521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8733629Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8733882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8733952Z return func(*args, **kwargs) 2025-12-04T09:47:16.8734221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8734343Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8734576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8734646Z return self.act(input) 2025-12-04T09:47:16.8734649Z 2025-12-04T09:47:16.8734770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8734980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8735041Z res = mod(**inputs) 2025-12-04T09:47:16.8735291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8735364Z outputs = self.model( 2025-12-04T09:47:16.8735609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8735686Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8735910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8736004Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8736246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8736314Z return func(*args, **kwargs) 2025-12-04T09:47:16.8736565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8736647Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8736651Z 2025-12-04T09:47:16.8736752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8736956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8737016Z res = mod(**inputs) 2025-12-04T09:47:16.8737260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8737332Z outputs = self.model( 2025-12-04T09:47:16.8737579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8737653Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8737878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8737954Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8738202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8738268Z return func(*args, **kwargs) 2025-12-04T09:47:16.8738516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.8738600Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.8738604Z 2025-12-04T09:47:16.8738704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8738904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8738967Z res = mod(**inputs) 2025-12-04T09:47:16.8739246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8739333Z outputs = self.model( 2025-12-04T09:47:16.8739573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8739667Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8739879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8739953Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8740191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8740257Z return func(*args, **kwargs) 2025-12-04T09:47:16.8740496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8740619Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8740860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8740935Z return func(*args, **kwargs) 2025-12-04T09:47:16.8741180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8741292Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8741295Z 2025-12-04T09:47:16.8741406Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8741599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8741680Z res = mod(**inputs) 2025-12-04T09:47:16.8741918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8741999Z outputs = self.model( 2025-12-04T09:47:16.8742245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8742316Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8742534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8742618Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8742859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8742933Z return func(*args, **kwargs) 2025-12-04T09:47:16.8743183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8743279Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8743535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8743602Z return func(*args, **kwargs) 2025-12-04T09:47:16.8743853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8743940Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8743945Z 2025-12-04T09:47:16.8744045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8744248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8744310Z res = mod(**inputs) 2025-12-04T09:47:16.8744563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8744637Z outputs = self.model( 2025-12-04T09:47:16.8744888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8744965Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8745211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8745290Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8745544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8745630Z return func(*args, **kwargs) 2025-12-04T09:47:16.8745874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8745980Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8746217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8746291Z return func(*args, **kwargs) 2025-12-04T09:47:16.8746538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8746663Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8746669Z 2025-12-04T09:47:16.8746781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8746975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8747047Z res = mod(**inputs) 2025-12-04T09:47:16.8747292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8747357Z outputs = self.model( 2025-12-04T09:47:16.8747610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8747681Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8747902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8748007Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8748249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8748323Z return func(*args, **kwargs) 2025-12-04T09:47:16.8748574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8748672Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8748915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8748981Z return func(*args, **kwargs) 2025-12-04T09:47:16.8749224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8749367Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8749370Z 2025-12-04T09:47:16.8749471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8749674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8749736Z res = mod(**inputs) 2025-12-04T09:47:16.8749981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8750055Z outputs = self.model( 2025-12-04T09:47:16.8750300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8750374Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8750591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8750666Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8750914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8750983Z return func(*args, **kwargs) 2025-12-04T09:47:16.8751243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8751348Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8751606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8751678Z return func(*args, **kwargs) 2025-12-04T09:47:16.8751926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8752010Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8752013Z 2025-12-04T09:47:16.8752119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8752314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8752381Z res = mod(**inputs) 2025-12-04T09:47:16.8752649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8752715Z outputs = self.model( 2025-12-04T09:47:16.8752967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8753039Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8753256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8753338Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8753583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8753656Z return func(*args, **kwargs) 2025-12-04T09:47:16.8753909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8754002Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8754242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8754307Z return func(*args, **kwargs) 2025-12-04T09:47:16.8754549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8754647Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8754650Z 2025-12-04T09:47:16.8754747Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8754940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8755000Z res = mod(**inputs) 2025-12-04T09:47:16.8755237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8755310Z outputs = self.model( 2025-12-04T09:47:16.8755551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8755627Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8755839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8755916Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8756157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8756223Z return func(*args, **kwargs) 2025-12-04T09:47:16.8756461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8756561Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8756794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8756868Z return func(*args, **kwargs) 2025-12-04T09:47:16.8757122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8757262Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8757265Z 2025-12-04T09:47:16.8757370Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8757555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8757623Z res = mod(**inputs) 2025-12-04T09:47:16.8757860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8757924Z outputs = self.model( 2025-12-04T09:47:16.8758172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8758239Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8758475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8758569Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8758806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8758877Z return func(*args, **kwargs) 2025-12-04T09:47:16.8759120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8759213Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8759524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8759614Z return func(*args, **kwargs) 2025-12-04T09:47:16.8759849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8759933Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8759937Z 2025-12-04T09:47:16.8760033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8760223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8760284Z res = mod(**inputs) 2025-12-04T09:47:16.8760517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8760588Z outputs = self.model( 2025-12-04T09:47:16.8760821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8760895Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8761105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8761179Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8761426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8761493Z return func(*args, **kwargs) 2025-12-04T09:47:16.8761734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8761855Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8761858Z 2025-12-04T09:47:16.8761956Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8762152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8762213Z res = mod(**inputs) 2025-12-04T09:47:16.8762453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8762527Z outputs = self.model( 2025-12-04T09:47:16.8762785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8762863Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8763075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8763166Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8763412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8763477Z return func(*args, **kwargs) 2025-12-04T09:47:16.8763718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8763836Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8764045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8764133Z return self.act(input) 2025-12-04T09:47:16.8764138Z 2025-12-04T09:47:16.8764239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8764429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8764501Z res = mod(**inputs) 2025-12-04T09:47:16.8764741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8764803Z outputs = self.model( 2025-12-04T09:47:16.8765053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8765122Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8765342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8765437Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8765685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8765761Z return func(*args, **kwargs) 2025-12-04T09:47:16.8766007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8766095Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8766099Z 2025-12-04T09:47:16.8766199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8766394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8766463Z res = mod(**inputs) 2025-12-04T09:47:16.8766710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8766778Z outputs = self.model( 2025-12-04T09:47:16.8767033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8767103Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8767331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8767408Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8767648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8767724Z return func(*args, **kwargs) 2025-12-04T09:47:16.8767970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8768065Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8768313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8768382Z return func(*args, **kwargs) 2025-12-04T09:47:16.8768654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8768764Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8768768Z 2025-12-04T09:47:16.8768885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8769092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8769155Z res = mod(**inputs) 2025-12-04T09:47:16.8769412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8769480Z outputs = self.model( 2025-12-04T09:47:16.8769727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8769808Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8770044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8770127Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8770377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8770447Z return func(*args, **kwargs) 2025-12-04T09:47:16.8770706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8770801Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8771040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8771114Z return func(*args, **kwargs) 2025-12-04T09:47:16.8771391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8771572Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8771579Z 2025-12-04T09:47:16.8771696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8771908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8771990Z res = mod(**inputs) 2025-12-04T09:47:16.8772420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8772504Z outputs = self.model( 2025-12-04T09:47:16.8772776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8772847Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8773078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8773159Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8773402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8773483Z return func(*args, **kwargs) 2025-12-04T09:47:16.8773731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8773828Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8774080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8774147Z return func(*args, **kwargs) 2025-12-04T09:47:16.8774399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8774508Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8774512Z 2025-12-04T09:47:16.8774617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8774821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8774884Z res = mod(**inputs) 2025-12-04T09:47:16.8775186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8775290Z outputs = self.model( 2025-12-04T09:47:16.8775539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8775617Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8775838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8775915Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8776168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8776238Z return func(*args, **kwargs) 2025-12-04T09:47:16.8776516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8776619Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8776859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8776936Z return func(*args, **kwargs) 2025-12-04T09:47:16.8777182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8777326Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8777329Z 2025-12-04T09:47:16.8777430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8777623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8777718Z res = mod(**inputs) 2025-12-04T09:47:16.8777966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8778033Z outputs = self.model( 2025-12-04T09:47:16.8778284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8778356Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8778581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8778658Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8778897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8778970Z return func(*args, **kwargs) 2025-12-04T09:47:16.8779213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8779308Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8779554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8779621Z return func(*args, **kwargs) 2025-12-04T09:47:16.8779871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8779959Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8779962Z 2025-12-04T09:47:16.8780061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8780259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8780321Z res = mod(**inputs) 2025-12-04T09:47:16.8780571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8780639Z outputs = self.model( 2025-12-04T09:47:16.8780883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8780977Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8781196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8781292Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8781540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8781608Z return func(*args, **kwargs) 2025-12-04T09:47:16.8781860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8781954Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8782195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8782271Z return func(*args, **kwargs) 2025-12-04T09:47:16.8782543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8782648Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8782651Z 2025-12-04T09:47:16.8782755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8782949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8783019Z res = mod(**inputs) 2025-12-04T09:47:16.8783266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8783331Z outputs = self.model( 2025-12-04T09:47:16.8783585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8783677Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8783906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8783983Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8784223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8784301Z return func(*args, **kwargs) 2025-12-04T09:47:16.8784548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8784642Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8784890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8784957Z return func(*args, **kwargs) 2025-12-04T09:47:16.8785210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8785338Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8785344Z 2025-12-04T09:47:16.8785445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8785646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8785711Z res = mod(**inputs) 2025-12-04T09:47:16.8785962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8786029Z outputs = self.model( 2025-12-04T09:47:16.8786273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8786353Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8786572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8786649Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8786898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8786982Z return func(*args, **kwargs) 2025-12-04T09:47:16.8787239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8787350Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8787588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8787665Z return func(*args, **kwargs) 2025-12-04T09:47:16.8787909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8787995Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8788000Z 2025-12-04T09:47:16.8788100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8788307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8788380Z res = mod(**inputs) 2025-12-04T09:47:16.8788625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8788694Z outputs = self.model( 2025-12-04T09:47:16.8788947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8789017Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8789242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8789317Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8789556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8789649Z return func(*args, **kwargs) 2025-12-04T09:47:16.8789895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8790015Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8790026Z 2025-12-04T09:47:16.8790127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8790321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8790390Z res = mod(**inputs) 2025-12-04T09:47:16.8790633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8790699Z outputs = self.model( 2025-12-04T09:47:16.8790950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8791022Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8791250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8791329Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8791569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8791646Z return func(*args, **kwargs) 2025-12-04T09:47:16.8791899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8792011Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8792229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8792295Z return self.act(input) 2025-12-04T09:47:16.8792299Z 2025-12-04T09:47:16.8792402Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8792592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8792653Z res = mod(**inputs) 2025-12-04T09:47:16.8792919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8792987Z outputs = self.model( 2025-12-04T09:47:16.8793242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8793317Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8793528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8793608Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8793842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8793909Z return func(*args, **kwargs) 2025-12-04T09:47:16.8794155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8794245Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8794251Z 2025-12-04T09:47:16.8794358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8794552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8794614Z res = mod(**inputs) 2025-12-04T09:47:16.8794861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8794924Z outputs = self.model( 2025-12-04T09:47:16.8795165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8795242Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8795473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8795551Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8795784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8795849Z return func(*args, **kwargs) 2025-12-04T09:47:16.8796094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.8796169Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.8796172Z 2025-12-04T09:47:16.8796275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8796460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8796519Z res = mod(**inputs) 2025-12-04T09:47:16.8796764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8796830Z outputs = self.model( 2025-12-04T09:47:16.8797066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8797145Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8797361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8797447Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8797685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8797753Z return func(*args, **kwargs) 2025-12-04T09:47:16.8798002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8798099Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8798339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8798415Z return func(*args, **kwargs) 2025-12-04T09:47:16.8798676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8798799Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8798820Z 2025-12-04T09:47:16.8798921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8799115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8799185Z res = mod(**inputs) 2025-12-04T09:47:16.8799441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8799516Z outputs = self.model( 2025-12-04T09:47:16.8799763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8799836Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8800085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8800163Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8800403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8800479Z return func(*args, **kwargs) 2025-12-04T09:47:16.8800724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8800830Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8801071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8801137Z return func(*args, **kwargs) 2025-12-04T09:47:16.8801413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8801492Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8801496Z 2025-12-04T09:47:16.8801604Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8801796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8801859Z res = mod(**inputs) 2025-12-04T09:47:16.8802110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8802174Z outputs = self.model( 2025-12-04T09:47:16.8802415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8802492Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8802712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8802795Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8803036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8803103Z return func(*args, **kwargs) 2025-12-04T09:47:16.8803356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8803451Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8803690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8803763Z return func(*args, **kwargs) 2025-12-04T09:47:16.8804008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8804121Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8804127Z 2025-12-04T09:47:16.8804228Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8804424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8804510Z res = mod(**inputs) 2025-12-04T09:47:16.8804761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8805451Z outputs = self.model( 2025-12-04T09:47:16.8805714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8805788Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8806028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8806108Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8806361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8806442Z return func(*args, **kwargs) 2025-12-04T09:47:16.8806728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8806842Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8807097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8807168Z return func(*args, **kwargs) 2025-12-04T09:47:16.8807433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8807572Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8807576Z 2025-12-04T09:47:16.8807686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8807887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8807977Z res = mod(**inputs) 2025-12-04T09:47:16.8808251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8808322Z outputs = self.model( 2025-12-04T09:47:16.8808583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8808668Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8808899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8808982Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8809237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8809307Z return func(*args, **kwargs) 2025-12-04T09:47:16.8809576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8809676Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8809926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8810002Z return func(*args, **kwargs) 2025-12-04T09:47:16.8810258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8810352Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8810356Z 2025-12-04T09:47:16.8810460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8810661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8810732Z res = mod(**inputs) 2025-12-04T09:47:16.8810993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8811070Z outputs = self.model( 2025-12-04T09:47:16.8811361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8811501Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8811755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8811859Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8812113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8812191Z return func(*args, **kwargs) 2025-12-04T09:47:16.8812450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8812557Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8812809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8812917Z return func(*args, **kwargs) 2025-12-04T09:47:16.8813189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8813289Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8813296Z 2025-12-04T09:47:16.8813410Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8813618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8813685Z res = mod(**inputs) 2025-12-04T09:47:16.8813951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8814021Z outputs = self.model( 2025-12-04T09:47:16.8814280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8814384Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8814631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8814719Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8814989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8815061Z return func(*args, **kwargs) 2025-12-04T09:47:16.8815347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8815449Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8815719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8815798Z return func(*args, **kwargs) 2025-12-04T09:47:16.8816073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8816216Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8816222Z 2025-12-04T09:47:16.8816329Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8816542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8816616Z res = mod(**inputs) 2025-12-04T09:47:16.8816890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8816966Z outputs = self.model( 2025-12-04T09:47:16.8817235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8817309Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8817557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8817640Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8817923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8818011Z return func(*args, **kwargs) 2025-12-04T09:47:16.8818289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8818395Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8818649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8818719Z return func(*args, **kwargs) 2025-12-04T09:47:16.8818984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8819069Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8819073Z 2025-12-04T09:47:16.8819184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8819412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8819480Z res = mod(**inputs) 2025-12-04T09:47:16.8819747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8819819Z outputs = self.model( 2025-12-04T09:47:16.8820073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8820158Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8820388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8820476Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8820731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8820825Z return func(*args, **kwargs) 2025-12-04T09:47:16.8821102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8821228Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8821233Z 2025-12-04T09:47:16.8821340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8821555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8821620Z res = mod(**inputs) 2025-12-04T09:47:16.8821897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8821967Z outputs = self.model( 2025-12-04T09:47:16.8822233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8822316Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8822556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8822645Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8822904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8822984Z return func(*args, **kwargs) 2025-12-04T09:47:16.8823242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8823357Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8823574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8823649Z return self.act(input) 2025-12-04T09:47:16.8823654Z 2025-12-04T09:47:16.8823755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8823961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8824022Z res = mod(**inputs) 2025-12-04T09:47:16.8824292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8824384Z outputs = self.model( 2025-12-04T09:47:16.8824626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8824696Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8824920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8824995Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8825240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8825308Z return func(*args, **kwargs) 2025-12-04T09:47:16.8825570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8825661Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8825665Z 2025-12-04T09:47:16.8825766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8825966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8826028Z res = mod(**inputs) 2025-12-04T09:47:16.8826270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8826341Z outputs = self.model( 2025-12-04T09:47:16.8826582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8826671Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8826905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8826982Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8827235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8827303Z return func(*args, **kwargs) 2025-12-04T09:47:16.8827555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8827659Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8827906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8827974Z return func(*args, **kwargs) 2025-12-04T09:47:16.8828231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8828344Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8828347Z 2025-12-04T09:47:16.8828455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8828658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8828721Z res = mod(**inputs) 2025-12-04T09:47:16.8828985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8829051Z outputs = self.model( 2025-12-04T09:47:16.8829307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8829376Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8829602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8829689Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8829937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8830004Z return func(*args, **kwargs) 2025-12-04T09:47:16.8830291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8830404Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8830648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8830716Z return func(*args, **kwargs) 2025-12-04T09:47:16.8830958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8831043Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8831046Z 2025-12-04T09:47:16.8831146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8831347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8831425Z res = mod(**inputs) 2025-12-04T09:47:16.8831674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8831748Z outputs = self.model( 2025-12-04T09:47:16.8831995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8832064Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8832293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8832368Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8832613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8832702Z return func(*args, **kwargs) 2025-12-04T09:47:16.8832949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8833055Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8833295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8833363Z return func(*args, **kwargs) 2025-12-04T09:47:16.8833617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8833725Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8833729Z 2025-12-04T09:47:16.8833834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8834029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8834093Z res = mod(**inputs) 2025-12-04T09:47:16.8834348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8834414Z outputs = self.model( 2025-12-04T09:47:16.8834666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8834735Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8834958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8835040Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8835280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8835346Z return func(*args, **kwargs) 2025-12-04T09:47:16.8835601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8835697Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8835944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8836027Z return func(*args, **kwargs) 2025-12-04T09:47:16.8836276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8836439Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8836442Z 2025-12-04T09:47:16.8836542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8836741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8836804Z res = mod(**inputs) 2025-12-04T09:47:16.8837044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8837116Z outputs = self.model( 2025-12-04T09:47:16.8837376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8837449Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8837683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8837761Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8838014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8838080Z return func(*args, **kwargs) 2025-12-04T09:47:16.8838329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8838431Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8838675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8838759Z return func(*args, **kwargs) 2025-12-04T09:47:16.8839012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8839096Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8839100Z 2025-12-04T09:47:16.8839207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8839400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8839461Z res = mod(**inputs) 2025-12-04T09:47:16.8839712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8839777Z outputs = self.model( 2025-12-04T09:47:16.8840031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8840102Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8840322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8840407Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8840646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8840716Z return func(*args, **kwargs) 2025-12-04T09:47:16.8840967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8841059Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8841305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8841372Z return func(*args, **kwargs) 2025-12-04T09:47:16.8841617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8841722Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8841725Z 2025-12-04T09:47:16.8841827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8842049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8842129Z res = mod(**inputs) 2025-12-04T09:47:16.8842379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8842450Z outputs = self.model( 2025-12-04T09:47:16.8842694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8842763Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8842988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8843065Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8843329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8843401Z return func(*args, **kwargs) 2025-12-04T09:47:16.8843646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8843751Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8843990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8844058Z return func(*args, **kwargs) 2025-12-04T09:47:16.8844312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8844438Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8844460Z 2025-12-04T09:47:16.8844570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8844767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8844830Z res = mod(**inputs) 2025-12-04T09:47:16.8845088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8845156Z outputs = self.model( 2025-12-04T09:47:16.8845410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8845480Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8845701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8845784Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8846026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8846097Z return func(*args, **kwargs) 2025-12-04T09:47:16.8846355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8846453Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8846700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8846770Z return func(*args, **kwargs) 2025-12-04T09:47:16.8847017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8847104Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8847107Z 2025-12-04T09:47:16.8847206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8847407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8847471Z res = mod(**inputs) 2025-12-04T09:47:16.8847720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8847791Z outputs = self.model( 2025-12-04T09:47:16.8848057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8848147Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8848373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8848450Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8848700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8848768Z return func(*args, **kwargs) 2025-12-04T09:47:16.8849014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8849142Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8849145Z 2025-12-04T09:47:16.8849265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8849470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8849532Z res = mod(**inputs) 2025-12-04T09:47:16.8849778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8849853Z outputs = self.model( 2025-12-04T09:47:16.8850099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8850169Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8850395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8850497Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8850748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8850816Z return func(*args, **kwargs) 2025-12-04T09:47:16.8851067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8851197Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8851508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8851594Z return self.act(input) 2025-12-04T09:47:16.8851599Z 2025-12-04T09:47:16.8851719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8851933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8852008Z res = mod(**inputs) 2025-12-04T09:47:16.8852281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8852352Z outputs = self.model( 2025-12-04T09:47:16.8852632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8852710Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8852963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8853046Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8853296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8853371Z return func(*args, **kwargs) 2025-12-04T09:47:16.8853618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8853699Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8853703Z 2025-12-04T09:47:16.8853811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8854024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8854096Z res = mod(**inputs) 2025-12-04T09:47:16.8854344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8854429Z outputs = self.model( 2025-12-04T09:47:16.8854679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8854748Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8854964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8855045Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8855284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8855360Z return func(*args, **kwargs) 2025-12-04T09:47:16.8855622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.8855702Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.8855707Z 2025-12-04T09:47:16.8855814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8856008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8856075Z res = mod(**inputs) 2025-12-04T09:47:16.8856324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8856389Z outputs = self.model( 2025-12-04T09:47:16.8856643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8856733Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8856956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8857044Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8857284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8857362Z return func(*args, **kwargs) 2025-12-04T09:47:16.8857611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8857707Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8857955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8858023Z return func(*args, **kwargs) 2025-12-04T09:47:16.8858267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8858387Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8858391Z 2025-12-04T09:47:16.8858495Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8858697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8858761Z res = mod(**inputs) 2025-12-04T09:47:16.8859011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8859084Z outputs = self.model( 2025-12-04T09:47:16.8859328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8859405Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8859626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8859705Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8859953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8860039Z return func(*args, **kwargs) 2025-12-04T09:47:16.8860285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8860407Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8860644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8860718Z return func(*args, **kwargs) 2025-12-04T09:47:16.8860961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8861038Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8861043Z 2025-12-04T09:47:16.8861150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8861359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8861431Z res = mod(**inputs) 2025-12-04T09:47:16.8861679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8861748Z outputs = self.model( 2025-12-04T09:47:16.8862002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8862072Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8862294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8862377Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8862620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8862712Z return func(*args, **kwargs) 2025-12-04T09:47:16.8862960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8863058Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8863303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8863372Z return func(*args, **kwargs) 2025-12-04T09:47:16.8863616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8863731Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8863735Z 2025-12-04T09:47:16.8863833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8864035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8864098Z res = mod(**inputs) 2025-12-04T09:47:16.8864345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8864422Z outputs = self.model( 2025-12-04T09:47:16.8864663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8864744Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8864964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8865042Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8865287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8865356Z return func(*args, **kwargs) 2025-12-04T09:47:16.8865600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8865704Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8865960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8866037Z return func(*args, **kwargs) 2025-12-04T09:47:16.8866284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8866435Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8866439Z 2025-12-04T09:47:16.8866548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8866750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8866819Z res = mod(**inputs) 2025-12-04T09:47:16.8867064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8867131Z outputs = self.model( 2025-12-04T09:47:16.8867395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8867470Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8867689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8867775Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8868011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8868082Z return func(*args, **kwargs) 2025-12-04T09:47:16.8868327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8868421Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8868685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8868750Z return func(*args, **kwargs) 2025-12-04T09:47:16.8868999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8869091Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8869096Z 2025-12-04T09:47:16.8869194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8869392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8869454Z res = mod(**inputs) 2025-12-04T09:47:16.8869698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8869769Z outputs = self.model( 2025-12-04T09:47:16.8870012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8870089Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8870309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8870387Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8870634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8870699Z return func(*args, **kwargs) 2025-12-04T09:47:16.8870935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8871035Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8871266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8871338Z return func(*args, **kwargs) 2025-12-04T09:47:16.8871576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8871679Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8871683Z 2025-12-04T09:47:16.8871800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8871987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8872073Z res = mod(**inputs) 2025-12-04T09:47:16.8872460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8872530Z outputs = self.model( 2025-12-04T09:47:16.8872773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8872842Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8873057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8873145Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8873446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8873523Z return func(*args, **kwargs) 2025-12-04T09:47:16.8873764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8873861Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8874099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8874164Z return func(*args, **kwargs) 2025-12-04T09:47:16.8874401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8874532Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8874559Z 2025-12-04T09:47:16.8874663Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8874867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8874932Z res = mod(**inputs) 2025-12-04T09:47:16.8875177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8875255Z outputs = self.model( 2025-12-04T09:47:16.8875501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8875580Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8875801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8875875Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8876129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8876197Z return func(*args, **kwargs) 2025-12-04T09:47:16.8876447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8876557Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8876798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8876869Z return func(*args, **kwargs) 2025-12-04T09:47:16.8877115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8877194Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8877198Z 2025-12-04T09:47:16.8877304Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8877496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8877566Z res = mod(**inputs) 2025-12-04T09:47:16.8877812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8877900Z outputs = self.model( 2025-12-04T09:47:16.8878156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8878252Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8878479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8878562Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8878806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8878879Z return func(*args, **kwargs) 2025-12-04T09:47:16.8879131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8879264Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8879269Z 2025-12-04T09:47:16.8879381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8879574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8879639Z res = mod(**inputs) 2025-12-04T09:47:16.8879890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8879957Z outputs = self.model( 2025-12-04T09:47:16.8880210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8880281Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8880501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8880605Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8880846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8880920Z return func(*args, **kwargs) 2025-12-04T09:47:16.8881168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8881284Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8881504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8881570Z return self.act(input) 2025-12-04T09:47:16.8881574Z 2025-12-04T09:47:16.8881673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8881878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8881941Z res = mod(**inputs) 2025-12-04T09:47:16.8882192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8882259Z outputs = self.model( 2025-12-04T09:47:16.8882505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8882584Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8882803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8882878Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8883123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8883190Z return func(*args, **kwargs) 2025-12-04T09:47:16.8883440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8883521Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8883525Z 2025-12-04T09:47:16.8883625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8883844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8883908Z res = mod(**inputs) 2025-12-04T09:47:16.8884176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8884242Z outputs = self.model( 2025-12-04T09:47:16.8884490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8884567Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8884784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8884859Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8885106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8885190Z return func(*args, **kwargs) 2025-12-04T09:47:16.8885442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8885540Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8885778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8885853Z return func(*args, **kwargs) 2025-12-04T09:47:16.8886094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8886210Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8886214Z 2025-12-04T09:47:16.8886313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8886526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8886597Z res = mod(**inputs) 2025-12-04T09:47:16.8886841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8886908Z outputs = self.model( 2025-12-04T09:47:16.8887161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8887231Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8887458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8887533Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8887774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8887848Z return func(*args, **kwargs) 2025-12-04T09:47:16.8888091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8888189Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8888444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8888516Z return func(*args, **kwargs) 2025-12-04T09:47:16.8888782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8888865Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8888868Z 2025-12-04T09:47:16.8888975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8889189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8889254Z res = mod(**inputs) 2025-12-04T09:47:16.8889522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8889593Z outputs = self.model( 2025-12-04T09:47:16.8889873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8889959Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8890210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8890290Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8890552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8890620Z return func(*args, **kwargs) 2025-12-04T09:47:16.8890889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8890992Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8891260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8891341Z return func(*args, **kwargs) 2025-12-04T09:47:16.8891666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8891795Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8891800Z 2025-12-04T09:47:16.8891907Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8892110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8892182Z res = mod(**inputs) 2025-12-04T09:47:16.8892492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8892559Z outputs = self.model( 2025-12-04T09:47:16.8892839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8892912Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8893141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8893217Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8893458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8893533Z return func(*args, **kwargs) 2025-12-04T09:47:16.8893780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8893875Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8894122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8894191Z return func(*args, **kwargs) 2025-12-04T09:47:16.8894445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8894578Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8894582Z 2025-12-04T09:47:16.8894684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8894887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8894950Z res = mod(**inputs) 2025-12-04T09:47:16.8895201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8895265Z outputs = self.model( 2025-12-04T09:47:16.8895508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8895587Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8895806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8895883Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8896143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8896234Z return func(*args, **kwargs) 2025-12-04T09:47:16.8896497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8896594Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8896841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8896914Z return func(*args, **kwargs) 2025-12-04T09:47:16.8897171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8897265Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8897269Z 2025-12-04T09:47:16.8897385Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8897581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8897652Z res = mod(**inputs) 2025-12-04T09:47:16.8897896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8897963Z outputs = self.model( 2025-12-04T09:47:16.8898219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8898288Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8898513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8898610Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8898859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8898931Z return func(*args, **kwargs) 2025-12-04T09:47:16.8899179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8899276Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8899523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8899589Z return func(*args, **kwargs) 2025-12-04T09:47:16.8899841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8899936Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8899939Z 2025-12-04T09:47:16.8900038Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8900244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8900308Z res = mod(**inputs) 2025-12-04T09:47:16.8900560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8900626Z outputs = self.model( 2025-12-04T09:47:16.8900873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8900949Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8901169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8901245Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8901492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8901560Z return func(*args, **kwargs) 2025-12-04T09:47:16.8901816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8901932Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8902174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8902266Z return func(*args, **kwargs) 2025-12-04T09:47:16.8902520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8902650Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8902654Z 2025-12-04T09:47:16.8902753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8902942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8903013Z res = mod(**inputs) 2025-12-04T09:47:16.8903253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8903339Z outputs = self.model( 2025-12-04T09:47:16.8903587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8903659Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8903881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8903955Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8904187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8904258Z return func(*args, **kwargs) 2025-12-04T09:47:16.8904496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8904606Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8904847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8904914Z return func(*args, **kwargs) 2025-12-04T09:47:16.8905159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8905238Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8905241Z 2025-12-04T09:47:16.8905337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8905533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8905595Z res = mod(**inputs) 2025-12-04T09:47:16.8905837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8905902Z outputs = self.model( 2025-12-04T09:47:16.8906139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8906216Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8906429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8906504Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8906743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8906805Z return func(*args, **kwargs) 2025-12-04T09:47:16.8907046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8907157Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8907161Z 2025-12-04T09:47:16.8907257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8907455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8907516Z res = mod(**inputs) 2025-12-04T09:47:16.8907778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8907846Z outputs = self.model( 2025-12-04T09:47:16.8908127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8908203Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8908419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8908494Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8908734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8908800Z return func(*args, **kwargs) 2025-12-04T09:47:16.8909043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8909169Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8909379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8909452Z return self.act(input) 2025-12-04T09:47:16.8909456Z 2025-12-04T09:47:16.8909555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8909742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8909810Z res = mod(**inputs) 2025-12-04T09:47:16.8910046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8910117Z outputs = self.model( 2025-12-04T09:47:16.8910354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8910443Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8910671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8910746Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8910991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8911056Z return func(*args, **kwargs) 2025-12-04T09:47:16.8911298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8911381Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8911385Z 2025-12-04T09:47:16.8911484Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8911675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8911742Z res = mod(**inputs) 2025-12-04T09:47:16.8911986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8912056Z outputs = self.model( 2025-12-04T09:47:16.8912292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8912363Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8912586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8912661Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8912897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8912969Z return func(*args, **kwargs) 2025-12-04T09:47:16.8913212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.8913295Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.8913301Z 2025-12-04T09:47:16.8913414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8913606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8913695Z res = mod(**inputs) 2025-12-04T09:47:16.8913936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8914006Z outputs = self.model( 2025-12-04T09:47:16.8914248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8914317Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8914540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8914615Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8914868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8914955Z return func(*args, **kwargs) 2025-12-04T09:47:16.8915190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8915290Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8915522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8915585Z return func(*args, **kwargs) 2025-12-04T09:47:16.8915830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8915933Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8915965Z 2025-12-04T09:47:16.8916070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8916258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8916319Z res = mod(**inputs) 2025-12-04T09:47:16.8916566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8916632Z outputs = self.model( 2025-12-04T09:47:16.8916871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8916948Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8917165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8917247Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8917492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8917557Z return func(*args, **kwargs) 2025-12-04T09:47:16.8917799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8917891Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8918116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8918188Z return func(*args, **kwargs) 2025-12-04T09:47:16.8918418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8918501Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8918504Z 2025-12-04T09:47:16.8918603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8918786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8918856Z res = mod(**inputs) 2025-12-04T09:47:16.8919091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8919161Z outputs = self.model( 2025-12-04T09:47:16.8919422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8919504Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8919723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8919794Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8920024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8920093Z return func(*args, **kwargs) 2025-12-04T09:47:16.8920331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8920429Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8920678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8920744Z return func(*args, **kwargs) 2025-12-04T09:47:16.8920987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8921090Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8921094Z 2025-12-04T09:47:16.8921194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8921381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8921440Z res = mod(**inputs) 2025-12-04T09:47:16.8921679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8921759Z outputs = self.model( 2025-12-04T09:47:16.8921992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8922067Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8922276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8922355Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8922581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8922645Z return func(*args, **kwargs) 2025-12-04T09:47:16.8922884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8922977Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8923199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8923271Z return func(*args, **kwargs) 2025-12-04T09:47:16.8923504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8923639Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8923644Z 2025-12-04T09:47:16.8923743Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8923929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8923996Z res = mod(**inputs) 2025-12-04T09:47:16.8924232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8924304Z outputs = self.model( 2025-12-04T09:47:16.8924538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8924606Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8924827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8924919Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8925154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8925244Z return func(*args, **kwargs) 2025-12-04T09:47:16.8925485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8925585Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8925818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8925883Z return func(*args, **kwargs) 2025-12-04T09:47:16.8926129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8926212Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8926232Z 2025-12-04T09:47:16.8926346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8926540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8926604Z res = mod(**inputs) 2025-12-04T09:47:16.8926855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8926922Z outputs = self.model( 2025-12-04T09:47:16.8927165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8927242Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8927460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8927562Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8927795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8927861Z return func(*args, **kwargs) 2025-12-04T09:47:16.8928105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8928198Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8928435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8928507Z return func(*args, **kwargs) 2025-12-04T09:47:16.8928748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8928849Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8928855Z 2025-12-04T09:47:16.8928952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8929146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8929218Z res = mod(**inputs) 2025-12-04T09:47:16.8929463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8929540Z outputs = self.model( 2025-12-04T09:47:16.8929781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8929851Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8930081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8930155Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8930395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8930471Z return func(*args, **kwargs) 2025-12-04T09:47:16.8930740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8930849Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8931090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8931177Z return func(*args, **kwargs) 2025-12-04T09:47:16.8931627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8931770Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8931774Z 2025-12-04T09:47:16.8931895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8932106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8932178Z res = mod(**inputs) 2025-12-04T09:47:16.8932503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8932581Z outputs = self.model( 2025-12-04T09:47:16.8932867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8932958Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8933194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8933278Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8933528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8933596Z return func(*args, **kwargs) 2025-12-04T09:47:16.8933852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8933969Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8934217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8934302Z return func(*args, **kwargs) 2025-12-04T09:47:16.8934598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8934690Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8934694Z 2025-12-04T09:47:16.8934794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8934988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8935057Z res = mod(**inputs) 2025-12-04T09:47:16.8935303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8935377Z outputs = self.model( 2025-12-04T09:47:16.8935623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8935696Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8935924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8936000Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8936239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8936313Z return func(*args, **kwargs) 2025-12-04T09:47:16.8936560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8936680Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8936685Z 2025-12-04T09:47:16.8936784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8936980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8937049Z res = mod(**inputs) 2025-12-04T09:47:16.8937321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8937408Z outputs = self.model( 2025-12-04T09:47:16.8937657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8937727Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8937957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8938031Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8938276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8938352Z return func(*args, **kwargs) 2025-12-04T09:47:16.8938615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8938740Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8938953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8939021Z return self.act(input) 2025-12-04T09:47:16.8939025Z 2025-12-04T09:47:16.8939131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8939321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8939383Z res = mod(**inputs) 2025-12-04T09:47:16.8939633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8939697Z outputs = self.model( 2025-12-04T09:47:16.8939984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8940057Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8940281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8940366Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8940610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8940682Z return func(*args, **kwargs) 2025-12-04T09:47:16.8940936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8941015Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8941018Z 2025-12-04T09:47:16.8941127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8941325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8941386Z res = mod(**inputs) 2025-12-04T09:47:16.8941647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8941715Z outputs = self.model( 2025-12-04T09:47:16.8941970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8942042Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8942265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8942348Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8942593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8942659Z return func(*args, **kwargs) 2025-12-04T09:47:16.8942916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8943013Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8943283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8943367Z return func(*args, **kwargs) 2025-12-04T09:47:16.8943611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8943726Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8943730Z 2025-12-04T09:47:16.8943831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8944038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8944096Z res = mod(**inputs) 2025-12-04T09:47:16.8944332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8944405Z outputs = self.model( 2025-12-04T09:47:16.8944652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8944724Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8944945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8945020Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8945254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8945319Z return func(*args, **kwargs) 2025-12-04T09:47:16.8945550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8945652Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8945899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8945971Z return func(*args, **kwargs) 2025-12-04T09:47:16.8946204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8946282Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8946285Z 2025-12-04T09:47:16.8946391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8946581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8946639Z res = mod(**inputs) 2025-12-04T09:47:16.8946880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8946943Z outputs = self.model( 2025-12-04T09:47:16.8947185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8947254Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8947471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8947552Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8947785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8947850Z return func(*args, **kwargs) 2025-12-04T09:47:16.8948098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8948191Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8948428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8948494Z return func(*args, **kwargs) 2025-12-04T09:47:16.8948734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8948865Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8948869Z 2025-12-04T09:47:16.8948968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8949181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8949243Z res = mod(**inputs) 2025-12-04T09:47:16.8949481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8949554Z outputs = self.model( 2025-12-04T09:47:16.8949791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8949860Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8950081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8950174Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8950423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8950490Z return func(*args, **kwargs) 2025-12-04T09:47:16.8950746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8950854Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8951105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8951182Z return func(*args, **kwargs) 2025-12-04T09:47:16.8951440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8951604Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8951608Z 2025-12-04T09:47:16.8951724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8951943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8952014Z res = mod(**inputs) 2025-12-04T09:47:16.8952283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8952350Z outputs = self.model( 2025-12-04T09:47:16.8952621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8952690Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8952906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8952986Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8953222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8953287Z return func(*args, **kwargs) 2025-12-04T09:47:16.8953537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8953630Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8953868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8953942Z return func(*args, **kwargs) 2025-12-04T09:47:16.8954176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8954263Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8954266Z 2025-12-04T09:47:16.8954361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8954555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8954614Z res = mod(**inputs) 2025-12-04T09:47:16.8954871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8954945Z outputs = self.model( 2025-12-04T09:47:16.8955199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8955270Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8955492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8955565Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8955807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8955873Z return func(*args, **kwargs) 2025-12-04T09:47:16.8956112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8956230Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8956468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8956542Z return func(*args, **kwargs) 2025-12-04T09:47:16.8956778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8956871Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8956875Z 2025-12-04T09:47:16.8956978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8957166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8957225Z res = mod(**inputs) 2025-12-04T09:47:16.8957500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8957565Z outputs = self.model( 2025-12-04T09:47:16.8957805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8957872Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8958080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8958161Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8958386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8958450Z return func(*args, **kwargs) 2025-12-04T09:47:16.8958691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8958785Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8959025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8959091Z return func(*args, **kwargs) 2025-12-04T09:47:16.8959328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8959457Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8959460Z 2025-12-04T09:47:16.8959557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8959755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8959817Z res = mod(**inputs) 2025-12-04T09:47:16.8960062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8960133Z outputs = self.model( 2025-12-04T09:47:16.8960370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8960441Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8960689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8960765Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8961016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8961081Z return func(*args, **kwargs) 2025-12-04T09:47:16.8961322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8961422Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8961663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8961738Z return func(*args, **kwargs) 2025-12-04T09:47:16.8962007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8962088Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8962092Z 2025-12-04T09:47:16.8962198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8962387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8962448Z res = mod(**inputs) 2025-12-04T09:47:16.8962694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8962758Z outputs = self.model( 2025-12-04T09:47:16.8963007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8963076Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8963308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8963391Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8963631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8963698Z return func(*args, **kwargs) 2025-12-04T09:47:16.8963948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8964063Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8964066Z 2025-12-04T09:47:16.8964171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8964363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8964424Z res = mod(**inputs) 2025-12-04T09:47:16.8964676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8964742Z outputs = self.model( 2025-12-04T09:47:16.8964995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8965065Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8965286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8965368Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8965608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8965674Z return func(*args, **kwargs) 2025-12-04T09:47:16.8965922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8966036Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8966257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8966325Z return self.act(input) 2025-12-04T09:47:16.8966347Z 2025-12-04T09:47:16.8966456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8966673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8966759Z res = mod(**inputs) 2025-12-04T09:47:16.8967027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8967095Z outputs = self.model( 2025-12-04T09:47:16.8967353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8967435Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8967668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8967750Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8968031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8968104Z return func(*args, **kwargs) 2025-12-04T09:47:16.8968373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8968460Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8968464Z 2025-12-04T09:47:16.8968572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8968787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8968851Z res = mod(**inputs) 2025-12-04T09:47:16.8969113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8969208Z outputs = self.model( 2025-12-04T09:47:16.8969479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8969561Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8969801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8969884Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8970156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8970226Z return func(*args, **kwargs) 2025-12-04T09:47:16.8970502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.8970583Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.8970587Z 2025-12-04T09:47:16.8970697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8970918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8970985Z res = mod(**inputs) 2025-12-04T09:47:16.8971259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8971337Z outputs = self.model( 2025-12-04T09:47:16.8971679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8971770Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8972004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8972085Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8972511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8972593Z return func(*args, **kwargs) 2025-12-04T09:47:16.8972861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8973007Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8973252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8973354Z return func(*args, **kwargs) 2025-12-04T09:47:16.8973608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8973717Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8973721Z 2025-12-04T09:47:16.8973831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8974029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8974102Z res = mod(**inputs) 2025-12-04T09:47:16.8974348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8974438Z outputs = self.model( 2025-12-04T09:47:16.8974695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8974768Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8974986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8975071Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8975310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8975385Z return func(*args, **kwargs) 2025-12-04T09:47:16.8975628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8975751Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8976001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8976070Z return func(*args, **kwargs) 2025-12-04T09:47:16.8976323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.8976403Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.8976407Z 2025-12-04T09:47:16.8976508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8976713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8976775Z res = mod(**inputs) 2025-12-04T09:47:16.8977017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8977092Z outputs = self.model( 2025-12-04T09:47:16.8977335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8977416Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8977646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8977721Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8977959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8978023Z return func(*args, **kwargs) 2025-12-04T09:47:16.8978262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8978365Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8978605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8978681Z return func(*args, **kwargs) 2025-12-04T09:47:16.8978925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.8979053Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.8979057Z 2025-12-04T09:47:16.8979183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8979380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8979450Z res = mod(**inputs) 2025-12-04T09:47:16.8979700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8979764Z outputs = self.model( 2025-12-04T09:47:16.8980017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8980089Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8980330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8980419Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8980662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8980737Z return func(*args, **kwargs) 2025-12-04T09:47:16.8980980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8981077Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8981327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8981393Z return func(*args, **kwargs) 2025-12-04T09:47:16.8981639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.8981798Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.8981804Z 2025-12-04T09:47:16.8981907Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8982107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8982171Z res = mod(**inputs) 2025-12-04T09:47:16.8982417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8982490Z outputs = self.model( 2025-12-04T09:47:16.8982734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8982810Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8983027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8983104Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8983350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8983418Z return func(*args, **kwargs) 2025-12-04T09:47:16.8983662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8983765Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8984001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8984073Z return func(*args, **kwargs) 2025-12-04T09:47:16.8984315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.8984398Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.8984402Z 2025-12-04T09:47:16.8984511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8984706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8984773Z res = mod(**inputs) 2025-12-04T09:47:16.8985036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8985130Z outputs = self.model( 2025-12-04T09:47:16.8985382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8985453Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8985674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8985757Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8985994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8986069Z return func(*args, **kwargs) 2025-12-04T09:47:16.8986330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8986432Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8986687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8986754Z return func(*args, **kwargs) 2025-12-04T09:47:16.8987015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.8987117Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.8987121Z 2025-12-04T09:47:16.8987219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8987420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8987503Z res = mod(**inputs) 2025-12-04T09:47:16.8987757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8987831Z outputs = self.model( 2025-12-04T09:47:16.8988076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8988154Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8988373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8988450Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8988695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8988760Z return func(*args, **kwargs) 2025-12-04T09:47:16.8989006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8989112Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8989352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8989427Z return func(*args, **kwargs) 2025-12-04T09:47:16.8989670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.8989798Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.8989801Z 2025-12-04T09:47:16.8989912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8990105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8990175Z res = mod(**inputs) 2025-12-04T09:47:16.8990422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8990490Z outputs = self.model( 2025-12-04T09:47:16.8990742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8990830Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8991052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8991156Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8991398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8991473Z return func(*args, **kwargs) 2025-12-04T09:47:16.8991719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.8991814Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.8992061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8992129Z return func(*args, **kwargs) 2025-12-04T09:47:16.8992391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.8992490Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.8992493Z 2025-12-04T09:47:16.8992597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8992797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8992860Z res = mod(**inputs) 2025-12-04T09:47:16.8993101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8993173Z outputs = self.model( 2025-12-04T09:47:16.8993413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8993509Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8993734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8993811Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8994059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8994136Z return func(*args, **kwargs) 2025-12-04T09:47:16.8994377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8994498Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8994502Z 2025-12-04T09:47:16.8994600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8994796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8994860Z res = mod(**inputs) 2025-12-04T09:47:16.8995100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8995171Z outputs = self.model( 2025-12-04T09:47:16.8995411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8995487Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8995704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8995777Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8996020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8996085Z return func(*args, **kwargs) 2025-12-04T09:47:16.8996327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.8996448Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.8996658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.8996746Z return self.act(input) 2025-12-04T09:47:16.8996751Z 2025-12-04T09:47:16.8996852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8997059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8997127Z res = mod(**inputs) 2025-12-04T09:47:16.8997368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8997431Z outputs = self.model( 2025-12-04T09:47:16.8997676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8997744Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8997965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8998057Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.8998294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.8998365Z return func(*args, **kwargs) 2025-12-04T09:47:16.8998606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.8998689Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.8998693Z 2025-12-04T09:47:16.8998790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.8998976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.8999042Z res = mod(**inputs) 2025-12-04T09:47:16.8999290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.8999373Z outputs = self.model( 2025-12-04T09:47:16.8999624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.8999693Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.8999913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.8999989Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9000226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9000297Z return func(*args, **kwargs) 2025-12-04T09:47:16.9000537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9000642Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9000874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9000941Z return func(*args, **kwargs) 2025-12-04T09:47:16.9001190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9001296Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9001301Z 2025-12-04T09:47:16.9001401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9001596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9001657Z res = mod(**inputs) 2025-12-04T09:47:16.9001905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9001968Z outputs = self.model( 2025-12-04T09:47:16.9002209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9002287Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9002542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9002618Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9002893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9002960Z return func(*args, **kwargs) 2025-12-04T09:47:16.9003208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9003301Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9003533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9003607Z return func(*args, **kwargs) 2025-12-04T09:47:16.9003846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.9003944Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.9003950Z 2025-12-04T09:47:16.9004051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9004244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9004313Z res = mod(**inputs) 2025-12-04T09:47:16.9004554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9004619Z outputs = self.model( 2025-12-04T09:47:16.9004864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9004934Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9005158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9005255Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9005491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9005564Z return func(*args, **kwargs) 2025-12-04T09:47:16.9005803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9005896Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9006136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9006200Z return func(*args, **kwargs) 2025-12-04T09:47:16.9006445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9006552Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9006556Z 2025-12-04T09:47:16.9006653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9006854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9006913Z res = mod(**inputs) 2025-12-04T09:47:16.9007159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9007225Z outputs = self.model( 2025-12-04T09:47:16.9007463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9007538Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9007754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9007827Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9008066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9008135Z return func(*args, **kwargs) 2025-12-04T09:47:16.9008414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9008515Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9008770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9008845Z return func(*args, **kwargs) 2025-12-04T09:47:16.9009089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.9009226Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.9009230Z 2025-12-04T09:47:16.9009330Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9009523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9009593Z res = mod(**inputs) 2025-12-04T09:47:16.9009854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9009923Z outputs = self.model( 2025-12-04T09:47:16.9010176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9010250Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9010480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9010558Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9010800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9010872Z return func(*args, **kwargs) 2025-12-04T09:47:16.9011135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9011239Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9011553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9011627Z return func(*args, **kwargs) 2025-12-04T09:47:16.9011881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.9011972Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.9011976Z 2025-12-04T09:47:16.9012084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9012297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9012363Z res = mod(**inputs) 2025-12-04T09:47:16.9012629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9012700Z outputs = self.model( 2025-12-04T09:47:16.9012960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9013044Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9013277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9013366Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9013609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9013674Z return func(*args, **kwargs) 2025-12-04T09:47:16.9013925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9014020Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9014259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9014334Z return func(*args, **kwargs) 2025-12-04T09:47:16.9014598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.9014721Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.9014725Z 2025-12-04T09:47:16.9014830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9015030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9015102Z res = mod(**inputs) 2025-12-04T09:47:16.9015351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9015419Z outputs = self.model( 2025-12-04T09:47:16.9015680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9015754Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9016008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9016087Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9016328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9016403Z return func(*args, **kwargs) 2025-12-04T09:47:16.9016650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9016744Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9016992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9017079Z return func(*args, **kwargs) 2025-12-04T09:47:16.9017338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.9017467Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.9017470Z 2025-12-04T09:47:16.9017571Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9017776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9017838Z res = mod(**inputs) 2025-12-04T09:47:16.9018095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9018159Z outputs = self.model( 2025-12-04T09:47:16.9018408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9018488Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9018714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9018791Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9019039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9019107Z return func(*args, **kwargs) 2025-12-04T09:47:16.9019367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9019463Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9019706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9019781Z return func(*args, **kwargs) 2025-12-04T09:47:16.9020031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.9020130Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.9020133Z 2025-12-04T09:47:16.9020231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9020436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9020506Z res = mod(**inputs) 2025-12-04T09:47:16.9020745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9020826Z outputs = self.model( 2025-12-04T09:47:16.9021065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9021133Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9021348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9021418Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9021646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9021732Z return func(*args, **kwargs) 2025-12-04T09:47:16.9021968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9022077Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9022090Z 2025-12-04T09:47:16.9022186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9022371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9022436Z res = mod(**inputs) 2025-12-04T09:47:16.9022670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9022731Z outputs = self.model( 2025-12-04T09:47:16.9022971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9023056Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9023278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9023353Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9023584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9023658Z return func(*args, **kwargs) 2025-12-04T09:47:16.9023893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9024003Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9024211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.9024274Z return self.act(input) 2025-12-04T09:47:16.9024279Z 2025-12-04T09:47:16.9024382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9024572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9024631Z res = mod(**inputs) 2025-12-04T09:47:16.9024877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9024942Z outputs = self.model( 2025-12-04T09:47:16.9025183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9025250Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9025459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9025538Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9025769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9025832Z return func(*args, **kwargs) 2025-12-04T09:47:16.9026093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.9026170Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.9026173Z 2025-12-04T09:47:16.9026294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9026480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9026540Z res = mod(**inputs) 2025-12-04T09:47:16.9026776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9026838Z outputs = self.model( 2025-12-04T09:47:16.9027071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9027147Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9027369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9027454Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9027683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9027749Z return func(*args, **kwargs) 2025-12-04T09:47:16.9027999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.9028075Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.9028078Z 2025-12-04T09:47:16.9028178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9028364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9028422Z res = mod(**inputs) 2025-12-04T09:47:16.9028689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9028752Z outputs = self.model( 2025-12-04T09:47:16.9028985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9029060Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9029276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9029353Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9029580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9029643Z return func(*args, **kwargs) 2025-12-04T09:47:16.9029880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9029974Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9030202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9030273Z return func(*args, **kwargs) 2025-12-04T09:47:16.9030505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9030617Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9030620Z 2025-12-04T09:47:16.9030714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9030897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9030965Z res = mod(**inputs) 2025-12-04T09:47:16.9031201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9031272Z outputs = self.model( 2025-12-04T09:47:16.9031507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9031575Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9031807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9031883Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9032127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9032199Z return func(*args, **kwargs) 2025-12-04T09:47:16.9032444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9032545Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9032771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9032837Z return func(*args, **kwargs) 2025-12-04T09:47:16.9033095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.9033173Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.9033177Z 2025-12-04T09:47:16.9033283Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9033475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9033538Z res = mod(**inputs) 2025-12-04T09:47:16.9033788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9033853Z outputs = self.model( 2025-12-04T09:47:16.9034094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9034172Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9034397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9034478Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9034707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9034771Z return func(*args, **kwargs) 2025-12-04T09:47:16.9035009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9035100Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9035325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9035395Z return func(*args, **kwargs) 2025-12-04T09:47:16.9035623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9035742Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9035745Z 2025-12-04T09:47:16.9035840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9036019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9036084Z res = mod(**inputs) 2025-12-04T09:47:16.9036311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9036377Z outputs = self.model( 2025-12-04T09:47:16.9036602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9036666Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9036878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9036950Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9037170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9037237Z return func(*args, **kwargs) 2025-12-04T09:47:16.9037495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9037610Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9037833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9037895Z return func(*args, **kwargs) 2025-12-04T09:47:16.9038129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.9038250Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.9038253Z 2025-12-04T09:47:16.9038356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9038535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9038607Z res = mod(**inputs) 2025-12-04T09:47:16.9038847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9038911Z outputs = self.model( 2025-12-04T09:47:16.9039136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9039210Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9039416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9039491Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9039710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9039789Z return func(*args, **kwargs) 2025-12-04T09:47:16.9040025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9040115Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9040338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9040409Z return func(*args, **kwargs) 2025-12-04T09:47:16.9040642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.9040724Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.9040727Z 2025-12-04T09:47:16.9040821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9041003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9041070Z res = mod(**inputs) 2025-12-04T09:47:16.9041298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9041366Z outputs = self.model( 2025-12-04T09:47:16.9041594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9041661Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9041874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9041945Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9042165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9042231Z return func(*args, **kwargs) 2025-12-04T09:47:16.9042458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9042554Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9042778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9042854Z return func(*args, **kwargs) 2025-12-04T09:47:16.9043091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.9043196Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.9043200Z 2025-12-04T09:47:16.9043299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9043479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9043537Z res = mod(**inputs) 2025-12-04T09:47:16.9043774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9043837Z outputs = self.model( 2025-12-04T09:47:16.9044067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9044157Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9044369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9044452Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9044683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9044744Z return func(*args, **kwargs) 2025-12-04T09:47:16.9044981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9045071Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9045297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9045387Z return func(*args, **kwargs) 2025-12-04T09:47:16.9045625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.9045755Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.9045759Z 2025-12-04T09:47:16.9045857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9046045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9046114Z res = mod(**inputs) 2025-12-04T09:47:16.9046350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9046423Z outputs = self.model( 2025-12-04T09:47:16.9046673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9046741Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9046958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9047033Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9047258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9047331Z return func(*args, **kwargs) 2025-12-04T09:47:16.9047560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9047658Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9047887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9047951Z return func(*args, **kwargs) 2025-12-04T09:47:16.9048190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.9048266Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.9048269Z 2025-12-04T09:47:16.9048380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9048581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9048658Z res = mod(**inputs) 2025-12-04T09:47:16.9048896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9048958Z outputs = self.model( 2025-12-04T09:47:16.9049190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9049264Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9049474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9049554Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9049803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9049870Z return func(*args, **kwargs) 2025-12-04T09:47:16.9050114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9050229Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9050233Z 2025-12-04T09:47:16.9050335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9050521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9050581Z res = mod(**inputs) 2025-12-04T09:47:16.9050821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9050885Z outputs = self.model( 2025-12-04T09:47:16.9051137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9051213Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9051480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9051572Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9051816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9051887Z return func(*args, **kwargs) 2025-12-04T09:47:16.9052159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9052282Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9052510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.9052593Z return self.act(input) 2025-12-04T09:47:16.9052597Z 2025-12-04T09:47:16.9052706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9052921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9052985Z res = mod(**inputs) 2025-12-04T09:47:16.9053240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9053314Z outputs = self.model( 2025-12-04T09:47:16.9053555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9053642Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9053851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9053923Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9054160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9054226Z return func(*args, **kwargs) 2025-12-04T09:47:16.9054479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.9054567Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.9054586Z 2025-12-04T09:47:16.9054685Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9054880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9054940Z res = mod(**inputs) 2025-12-04T09:47:16.9055180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9055252Z outputs = self.model( 2025-12-04T09:47:16.9055489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9055560Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9055800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9055876Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9056120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9056188Z return func(*args, **kwargs) 2025-12-04T09:47:16.9056429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9056534Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9056777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9056847Z return func(*args, **kwargs) 2025-12-04T09:47:16.9057100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9057207Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9057212Z 2025-12-04T09:47:16.9057318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9057502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9057563Z res = mod(**inputs) 2025-12-04T09:47:16.9057803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9057865Z outputs = self.model( 2025-12-04T09:47:16.9058109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9058175Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9058385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9058468Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9058699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9058762Z return func(*args, **kwargs) 2025-12-04T09:47:16.9059002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9059100Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9059336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9059399Z return func(*args, **kwargs) 2025-12-04T09:47:16.9059633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.9059718Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.9059723Z 2025-12-04T09:47:16.9060053Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9060263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9060325Z res = mod(**inputs) 2025-12-04T09:47:16.9060559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9060650Z outputs = self.model( 2025-12-04T09:47:16.9060885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9060960Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9061171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9061246Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9061485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9061552Z return func(*args, **kwargs) 2025-12-04T09:47:16.9061803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9061911Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9062137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9062205Z return func(*args, **kwargs) 2025-12-04T09:47:16.9062438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9062542Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9062545Z 2025-12-04T09:47:16.9062647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9062831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9062908Z res = mod(**inputs) 2025-12-04T09:47:16.9063151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9063215Z outputs = self.model( 2025-12-04T09:47:16.9063453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9063524Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9063732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9063812Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9064038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9064112Z return func(*args, **kwargs) 2025-12-04T09:47:16.9064345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9064435Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9064666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9064730Z return func(*args, **kwargs) 2025-12-04T09:47:16.9064963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.9065094Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.9065098Z 2025-12-04T09:47:16.9065193Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9065385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9065444Z res = mod(**inputs) 2025-12-04T09:47:16.9065676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9065749Z outputs = self.model( 2025-12-04T09:47:16.9065997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9066072Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9066283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9066374Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9066611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9066673Z return func(*args, **kwargs) 2025-12-04T09:47:16.9066908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9067005Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9067236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9067319Z return func(*args, **kwargs) 2025-12-04T09:47:16.9067556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.9067638Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.9067642Z 2025-12-04T09:47:16.9067744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9067926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9067984Z res = mod(**inputs) 2025-12-04T09:47:16.9068222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9068284Z outputs = self.model( 2025-12-04T09:47:16.9068522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9068611Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9068821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9068901Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9069128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9069202Z return func(*args, **kwargs) 2025-12-04T09:47:16.9069435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9069523Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9069756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9069818Z return func(*args, **kwargs) 2025-12-04T09:47:16.9070050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.9070148Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.9070153Z 2025-12-04T09:47:16.9070246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9070436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9070495Z res = mod(**inputs) 2025-12-04T09:47:16.9070727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9070800Z outputs = self.model( 2025-12-04T09:47:16.9071033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9071107Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9071313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9071388Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9071641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9071707Z return func(*args, **kwargs) 2025-12-04T09:47:16.9071966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9072064Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9072476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9072556Z return func(*args, **kwargs) 2025-12-04T09:47:16.9072790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.9072918Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.9072921Z 2025-12-04T09:47:16.9073024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9073250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9073313Z res = mod(**inputs) 2025-12-04T09:47:16.9073557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9073622Z outputs = self.model( 2025-12-04T09:47:16.9073865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9073933Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9074141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9074222Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9074477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9074549Z return func(*args, **kwargs) 2025-12-04T09:47:16.9074784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9074875Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9075116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9075179Z return func(*args, **kwargs) 2025-12-04T09:47:16.9075412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.9075495Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.9075498Z 2025-12-04T09:47:16.9075594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9075787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9075847Z res = mod(**inputs) 2025-12-04T09:47:16.9076082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9076152Z outputs = self.model( 2025-12-04T09:47:16.9076384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9076451Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9076665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9076739Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9076972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9077035Z return func(*args, **kwargs) 2025-12-04T09:47:16.9077270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9077389Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9077392Z 2025-12-04T09:47:16.9077508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9077701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9077785Z res = mod(**inputs) 2025-12-04T09:47:16.9078023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9078092Z outputs = self.model( 2025-12-04T09:47:16.9078332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9078400Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9078619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9078693Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9078952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9079018Z return func(*args, **kwargs) 2025-12-04T09:47:16.9079254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9079374Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9079578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.9079647Z return self.act(input) 2025-12-04T09:47:16.9079650Z 2025-12-04T09:47:16.9079745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9079929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9080016Z res = mod(**inputs) 2025-12-04T09:47:16.9080250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9080313Z outputs = self.model( 2025-12-04T09:47:16.9080551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9080618Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9080831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9080902Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9081131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9081200Z return func(*args, **kwargs) 2025-12-04T09:47:16.9081432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.9081508Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.9081518Z 2025-12-04T09:47:16.9081616Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9081799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9081864Z res = mod(**inputs) 2025-12-04T09:47:16.9082095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9082155Z outputs = self.model( 2025-12-04T09:47:16.9082399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9082464Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9082675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9082748Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9082977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9083064Z return func(*args, **kwargs) 2025-12-04T09:47:16.9083298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.9083394Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.9083397Z 2025-12-04T09:47:16.9083500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9083687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9083751Z res = mod(**inputs) 2025-12-04T09:47:16.9083985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9084048Z outputs = self.model( 2025-12-04T09:47:16.9084288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9084371Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9084582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9084660Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9084888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9084960Z return func(*args, **kwargs) 2025-12-04T09:47:16.9085192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9085284Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9085517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9085602Z return func(*args, **kwargs) 2025-12-04T09:47:16.9085852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9085959Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9085963Z 2025-12-04T09:47:16.9086061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9086259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9086320Z res = mod(**inputs) 2025-12-04T09:47:16.9086557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9086627Z outputs = self.model( 2025-12-04T09:47:16.9086865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9086942Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9087156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9087231Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9087474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9087539Z return func(*args, **kwargs) 2025-12-04T09:47:16.9087786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9087880Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9088110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9088182Z return func(*args, **kwargs) 2025-12-04T09:47:16.9088416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.9088496Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.9088500Z 2025-12-04T09:47:16.9088606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9088808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9088878Z res = mod(**inputs) 2025-12-04T09:47:16.9089132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9089196Z outputs = self.model( 2025-12-04T09:47:16.9089438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9089506Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9089721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9089802Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9090036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9090124Z return func(*args, **kwargs) 2025-12-04T09:47:16.9090366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9116334Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9116799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9116883Z return func(*args, **kwargs) 2025-12-04T09:47:16.9117165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9117287Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9117295Z 2025-12-04T09:47:16.9117411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9117746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9117817Z res = mod(**inputs) 2025-12-04T09:47:16.9118099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9118176Z outputs = self.model( 2025-12-04T09:47:16.9118436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9118522Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9118759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9118851Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9119121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9119192Z return func(*args, **kwargs) 2025-12-04T09:47:16.9119444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9119550Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9119792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9119869Z return func(*args, **kwargs) 2025-12-04T09:47:16.9120112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.9120256Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.9120261Z 2025-12-04T09:47:16.9120367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9120569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9120640Z res = mod(**inputs) 2025-12-04T09:47:16.9120887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9120957Z outputs = self.model( 2025-12-04T09:47:16.9121247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9121320Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9121584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9121670Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9121909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9121989Z return func(*args, **kwargs) 2025-12-04T09:47:16.9122233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9122344Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9122611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9122685Z return func(*args, **kwargs) 2025-12-04T09:47:16.9122938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.9123029Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.9123033Z 2025-12-04T09:47:16.9123143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9123351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9123417Z res = mod(**inputs) 2025-12-04T09:47:16.9123671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9123741Z outputs = self.model( 2025-12-04T09:47:16.9124009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9124089Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9124307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9124391Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9124630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9124694Z return func(*args, **kwargs) 2025-12-04T09:47:16.9124939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9125030Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9125266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9125341Z return func(*args, **kwargs) 2025-12-04T09:47:16.9125582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.9125683Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.9125687Z 2025-12-04T09:47:16.9125784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9125982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9126053Z res = mod(**inputs) 2025-12-04T09:47:16.9126300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9126367Z outputs = self.model( 2025-12-04T09:47:16.9126618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9126689Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9126930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9127009Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9127262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9127354Z return func(*args, **kwargs) 2025-12-04T09:47:16.9127595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9127697Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9127935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9128000Z return func(*args, **kwargs) 2025-12-04T09:47:16.9128249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.9128379Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.9128382Z 2025-12-04T09:47:16.9128499Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9128703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9128767Z res = mod(**inputs) 2025-12-04T09:47:16.9129018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9129086Z outputs = self.model( 2025-12-04T09:47:16.9129326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9129407Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9129621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9129723Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9129964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9130029Z return func(*args, **kwargs) 2025-12-04T09:47:16.9130283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9130377Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9130616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9130689Z return func(*args, **kwargs) 2025-12-04T09:47:16.9130936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.9131025Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.9131028Z 2025-12-04T09:47:16.9131130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9131330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9131401Z res = mod(**inputs) 2025-12-04T09:47:16.9131757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9131830Z outputs = self.model( 2025-12-04T09:47:16.9132139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9132216Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9132460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9132541Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9132799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9132883Z return func(*args, **kwargs) 2025-12-04T09:47:16.9133146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9133305Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9133310Z 2025-12-04T09:47:16.9133421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9133662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9133742Z res = mod(**inputs) 2025-12-04T09:47:16.9134013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9134081Z outputs = self.model( 2025-12-04T09:47:16.9134338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9134408Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9134637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9134731Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9134976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9135055Z return func(*args, **kwargs) 2025-12-04T09:47:16.9135308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9135432Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9135650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.9135720Z return self.act(input) 2025-12-04T09:47:16.9135724Z 2025-12-04T09:47:16.9135834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9136048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9136110Z res = mod(**inputs) 2025-12-04T09:47:16.9136368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9136435Z outputs = self.model( 2025-12-04T09:47:16.9136687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9136762Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9136979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9137064Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9137306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9137374Z return func(*args, **kwargs) 2025-12-04T09:47:16.9137690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.9137773Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.9137776Z 2025-12-04T09:47:16.9137888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9138082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9138145Z res = mod(**inputs) 2025-12-04T09:47:16.9138397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9138463Z outputs = self.model( 2025-12-04T09:47:16.9138715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9138786Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9139004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9139091Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9139347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9139416Z return func(*args, **kwargs) 2025-12-04T09:47:16.9139666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9139785Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9140032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9140099Z return func(*args, **kwargs) 2025-12-04T09:47:16.9140346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9140464Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9140470Z 2025-12-04T09:47:16.9140570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9140790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9140857Z res = mod(**inputs) 2025-12-04T09:47:16.9141106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9141182Z outputs = self.model( 2025-12-04T09:47:16.9141431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9141502Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9141732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9141811Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9142064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9142149Z return func(*args, **kwargs) 2025-12-04T09:47:16.9142407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9142504Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9142743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9142820Z return func(*args, **kwargs) 2025-12-04T09:47:16.9143066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.9143154Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.9143158Z 2025-12-04T09:47:16.9143259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9143452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9143521Z res = mod(**inputs) 2025-12-04T09:47:16.9143769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9143838Z outputs = self.model( 2025-12-04T09:47:16.9144088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9144160Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9144386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9144463Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9144709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9144782Z return func(*args, **kwargs) 2025-12-04T09:47:16.9145027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9145127Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9145395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9145464Z return func(*args, **kwargs) 2025-12-04T09:47:16.9145736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9145842Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9145846Z 2025-12-04T09:47:16.9145941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9146137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9146197Z res = mod(**inputs) 2025-12-04T09:47:16.9146442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9146508Z outputs = self.model( 2025-12-04T09:47:16.9146757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9146839Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9147050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9147123Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9147362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9147425Z return func(*args, **kwargs) 2025-12-04T09:47:16.9147669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9147759Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9148002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9148075Z return func(*args, **kwargs) 2025-12-04T09:47:16.9148314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.9148449Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.9148454Z 2025-12-04T09:47:16.9148551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9148735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9148802Z res = mod(**inputs) 2025-12-04T09:47:16.9149038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9149102Z outputs = self.model( 2025-12-04T09:47:16.9149346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9149415Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9149631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9149704Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9149928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9149997Z return func(*args, **kwargs) 2025-12-04T09:47:16.9150224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9150310Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9150540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9150601Z return func(*args, **kwargs) 2025-12-04T09:47:16.9150837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.9150917Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.9150921Z 2025-12-04T09:47:16.9151034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9151230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9151310Z res = mod(**inputs) 2025-12-04T09:47:16.9151549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9151612Z outputs = self.model( 2025-12-04T09:47:16.9151840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9151913Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9152122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9152197Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9152449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9152515Z return func(*args, **kwargs) 2025-12-04T09:47:16.9152758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9152852Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9153081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9153151Z return func(*args, **kwargs) 2025-12-04T09:47:16.9153387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.9153483Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.9153502Z 2025-12-04T09:47:16.9153602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9153789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9153857Z res = mod(**inputs) 2025-12-04T09:47:16.9154092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9154156Z outputs = self.model( 2025-12-04T09:47:16.9154394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9154461Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9154678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9154754Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9154984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9155057Z return func(*args, **kwargs) 2025-12-04T09:47:16.9155294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9155385Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9155623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9155687Z return func(*args, **kwargs) 2025-12-04T09:47:16.9155929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.9156053Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.9156056Z 2025-12-04T09:47:16.9156153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9156346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9156408Z res = mod(**inputs) 2025-12-04T09:47:16.9156679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9156745Z outputs = self.model( 2025-12-04T09:47:16.9156979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9157069Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9157278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9157352Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9157584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9157649Z return func(*args, **kwargs) 2025-12-04T09:47:16.9157891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9157999Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9158231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9158301Z return func(*args, **kwargs) 2025-12-04T09:47:16.9158538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.9158621Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.9158625Z 2025-12-04T09:47:16.9158720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9158904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9158970Z res = mod(**inputs) 2025-12-04T09:47:16.9159202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9159285Z outputs = self.model( 2025-12-04T09:47:16.9159526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9159594Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9159806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9159881Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9160105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9160170Z return func(*args, **kwargs) 2025-12-04T09:47:16.9160398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9160507Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9160518Z 2025-12-04T09:47:16.9160612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9160795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9160856Z res = mod(**inputs) 2025-12-04T09:47:16.9161089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9161150Z outputs = self.model( 2025-12-04T09:47:16.9161383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9161447Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9161657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9161728Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9161954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9162024Z return func(*args, **kwargs) 2025-12-04T09:47:16.9162276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9162388Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9162624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.9162689Z return self.act(input) 2025-12-04T09:47:16.9162692Z 2025-12-04T09:47:16.9162791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9162976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9163035Z res = mod(**inputs) 2025-12-04T09:47:16.9163272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9163335Z outputs = self.model( 2025-12-04T09:47:16.9163580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9163657Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9163867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9163952Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9164183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9164247Z return func(*args, **kwargs) 2025-12-04T09:47:16.9164489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.9164565Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.9164569Z 2025-12-04T09:47:16.9164670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9164875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9164933Z res = mod(**inputs) 2025-12-04T09:47:16.9165170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9165231Z outputs = self.model( 2025-12-04T09:47:16.9165460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9165528Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9165735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9165813Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9166042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9166108Z return func(*args, **kwargs) 2025-12-04T09:47:16.9166350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.9166426Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.9166430Z 2025-12-04T09:47:16.9166530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9166716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9166772Z res = mod(**inputs) 2025-12-04T09:47:16.9167012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9167073Z outputs = self.model( 2025-12-04T09:47:16.9167305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9167375Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9167584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9167662Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9167910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9167976Z return func(*args, **kwargs) 2025-12-04T09:47:16.9168233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9168326Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9168551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9168621Z return func(*args, **kwargs) 2025-12-04T09:47:16.9168853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9168965Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9168969Z 2025-12-04T09:47:16.9169063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9169264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9169336Z res = mod(**inputs) 2025-12-04T09:47:16.9169575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9169649Z outputs = self.model( 2025-12-04T09:47:16.9169886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9169956Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9170176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9170250Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9170497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9170572Z return func(*args, **kwargs) 2025-12-04T09:47:16.9170811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9170913Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9171149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9171218Z return func(*args, **kwargs) 2025-12-04T09:47:16.9171574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.9171666Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.9171670Z 2025-12-04T09:47:16.9171779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9171999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9172064Z res = mod(**inputs) 2025-12-04T09:47:16.9172664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9172736Z outputs = self.model( 2025-12-04T09:47:16.9172978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9173058Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9173294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9173382Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9173635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9173704Z return func(*args, **kwargs) 2025-12-04T09:47:16.9173976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9174080Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9174413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9174522Z return func(*args, **kwargs) 2025-12-04T09:47:16.9174785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9174909Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9174913Z 2025-12-04T09:47:16.9175021Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9175229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9175305Z res = mod(**inputs) 2025-12-04T09:47:16.9175571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9175649Z outputs = self.model( 2025-12-04T09:47:16.9175941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9176019Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9176261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9176342Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9176596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9176675Z return func(*args, **kwargs) 2025-12-04T09:47:16.9176931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9177038Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9177331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9177404Z return func(*args, **kwargs) 2025-12-04T09:47:16.9177679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.9177824Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.9177831Z 2025-12-04T09:47:16.9177945Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9178152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9178216Z res = mod(**inputs) 2025-12-04T09:47:16.9178485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9178555Z outputs = self.model( 2025-12-04T09:47:16.9178816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9178901Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9179136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9179224Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9179483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9179553Z return func(*args, **kwargs) 2025-12-04T09:47:16.9179819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9179923Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9180150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9180219Z return func(*args, **kwargs) 2025-12-04T09:47:16.9180453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.9180554Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.9180558Z 2025-12-04T09:47:16.9180657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9180858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9180927Z res = mod(**inputs) 2025-12-04T09:47:16.9181163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9181233Z outputs = self.model( 2025-12-04T09:47:16.9181468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9181536Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9181755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9181848Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9182074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9182145Z return func(*args, **kwargs) 2025-12-04T09:47:16.9182375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9182470Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9182694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9182758Z return func(*args, **kwargs) 2025-12-04T09:47:16.9182991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.9183096Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.9183100Z 2025-12-04T09:47:16.9183195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9183388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9183448Z res = mod(**inputs) 2025-12-04T09:47:16.9183689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9183751Z outputs = self.model( 2025-12-04T09:47:16.9183981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9184054Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9184262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9184340Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9184570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9184632Z return func(*args, **kwargs) 2025-12-04T09:47:16.9184871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9184960Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9185188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9185256Z return func(*args, **kwargs) 2025-12-04T09:47:16.9185487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.9185613Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.9185617Z 2025-12-04T09:47:16.9185713Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9185898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9185966Z res = mod(**inputs) 2025-12-04T09:47:16.9186215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9186288Z outputs = self.model( 2025-12-04T09:47:16.9186538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9186605Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9186820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9186893Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9187121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9187195Z return func(*args, **kwargs) 2025-12-04T09:47:16.9187426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9187536Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9187765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9187829Z return func(*args, **kwargs) 2025-12-04T09:47:16.9188066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.9188145Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.9188148Z 2025-12-04T09:47:16.9188243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9188434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9188493Z res = mod(**inputs) 2025-12-04T09:47:16.9188751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9188816Z outputs = self.model( 2025-12-04T09:47:16.9189045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9189119Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9189327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9189406Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9189633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9189695Z return func(*args, **kwargs) 2025-12-04T09:47:16.9189934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9190046Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9190049Z 2025-12-04T09:47:16.9190145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9190337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9190397Z res = mod(**inputs) 2025-12-04T09:47:16.9190637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9190700Z outputs = self.model( 2025-12-04T09:47:16.9190930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9191003Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9191207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9191279Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9191515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9191579Z return func(*args, **kwargs) 2025-12-04T09:47:16.9191830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9191941Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9192161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.9192234Z return self.act(input) 2025-12-04T09:47:16.9192238Z 2025-12-04T09:47:16.9192333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9192523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9192581Z res = mod(**inputs) 2025-12-04T09:47:16.9192811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9192884Z outputs = self.model( 2025-12-04T09:47:16.9193131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9193198Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9193414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9193487Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9193718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9193780Z return func(*args, **kwargs) 2025-12-04T09:47:16.9194006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.9194091Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.9194109Z 2025-12-04T09:47:16.9194207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9194411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9194473Z res = mod(**inputs) 2025-12-04T09:47:16.9194708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9194778Z outputs = self.model( 2025-12-04T09:47:16.9195012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9195083Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9195292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9195363Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9195597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9195661Z return func(*args, **kwargs) 2025-12-04T09:47:16.9195901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9195993Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9196223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9196294Z return func(*args, **kwargs) 2025-12-04T09:47:16.9196527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9196630Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9196641Z 2025-12-04T09:47:16.9196736Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9196926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9196996Z res = mod(**inputs) 2025-12-04T09:47:16.9197238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9197318Z outputs = self.model( 2025-12-04T09:47:16.9197566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9197652Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9197871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9197946Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9198176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9198248Z return func(*args, **kwargs) 2025-12-04T09:47:16.9198485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9198580Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9198844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9198914Z return func(*args, **kwargs) 2025-12-04T09:47:16.9199162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.9199240Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.9199244Z 2025-12-04T09:47:16.9199341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9199537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9199598Z res = mod(**inputs) 2025-12-04T09:47:16.9199845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9199938Z outputs = self.model( 2025-12-04T09:47:16.9200187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9200265Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9200484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9200563Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9200809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9200875Z return func(*args, **kwargs) 2025-12-04T09:47:16.9201126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9201220Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9201456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9201531Z return func(*args, **kwargs) 2025-12-04T09:47:16.9201778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9201887Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9201899Z 2025-12-04T09:47:16.9202009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9202200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9202266Z res = mod(**inputs) 2025-12-04T09:47:16.9202506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9202569Z outputs = self.model( 2025-12-04T09:47:16.9202811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9202880Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9203101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9203209Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9203453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9203543Z return func(*args, **kwargs) 2025-12-04T09:47:16.9203781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9203874Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9204115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9204180Z return func(*args, **kwargs) 2025-12-04T09:47:16.9204425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.9204571Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.9204576Z 2025-12-04T09:47:16.9204676Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9204880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9204944Z res = mod(**inputs) 2025-12-04T09:47:16.9205184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9205255Z outputs = self.model( 2025-12-04T09:47:16.9205499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9205576Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9205796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9205893Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9206146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9206215Z return func(*args, **kwargs) 2025-12-04T09:47:16.9206468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9206565Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9206807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9206882Z return func(*args, **kwargs) 2025-12-04T09:47:16.9207129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.9207215Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.9207228Z 2025-12-04T09:47:16.9207332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9207530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9207601Z res = mod(**inputs) 2025-12-04T09:47:16.9207859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9207928Z outputs = self.model( 2025-12-04T09:47:16.9208183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9208252Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9208477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9208553Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9208794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9208871Z return func(*args, **kwargs) 2025-12-04T09:47:16.9209139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9209238Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9209483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9209572Z return func(*args, **kwargs) 2025-12-04T09:47:16.9209826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.9209921Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.9209925Z 2025-12-04T09:47:16.9210024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9210230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9210298Z res = mod(**inputs) 2025-12-04T09:47:16.9210578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9210661Z outputs = self.model( 2025-12-04T09:47:16.9210927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9211010Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9211244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9211326Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9211676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9211755Z return func(*args, **kwargs) 2025-12-04T09:47:16.9212028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9212153Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9212408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9212486Z return func(*args, **kwargs) 2025-12-04T09:47:16.9212744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.9212877Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.9212889Z 2025-12-04T09:47:16.9212994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9213198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9213274Z res = mod(**inputs) 2025-12-04T09:47:16.9213542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9213608Z outputs = self.model( 2025-12-04T09:47:16.9213858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9213927Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9214146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9214222Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9214452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9214521Z return func(*args, **kwargs) 2025-12-04T09:47:16.9214758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9214847Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9215086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9215152Z return func(*args, **kwargs) 2025-12-04T09:47:16.9215415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.9215499Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.9215520Z 2025-12-04T09:47:16.9215622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9215826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9215888Z res = mod(**inputs) 2025-12-04T09:47:16.9216139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9216210Z outputs = self.model( 2025-12-04T09:47:16.9216446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9216521Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9216749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9216828Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9217069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9217139Z return func(*args, **kwargs) 2025-12-04T09:47:16.9217392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9217508Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9217512Z 2025-12-04T09:47:16.9217612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9217816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9217904Z res = mod(**inputs) 2025-12-04T09:47:16.9218153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9218229Z outputs = self.model( 2025-12-04T09:47:16.9218473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9218553Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9218774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9218851Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9219096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9219164Z return func(*args, **kwargs) 2025-12-04T09:47:16.9219407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9219530Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9219748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.9219824Z return self.act(input) 2025-12-04T09:47:16.9219827Z 2025-12-04T09:47:16.9219928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9220127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9220197Z res = mod(**inputs) 2025-12-04T09:47:16.9220442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9220515Z outputs = self.model( 2025-12-04T09:47:16.9220757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9220830Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9221059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9221153Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9221395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9221500Z return func(*args, **kwargs) 2025-12-04T09:47:16.9221751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.9221841Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.9221846Z 2025-12-04T09:47:16.9221952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9222149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9222223Z res = mod(**inputs) 2025-12-04T09:47:16.9222479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9222555Z outputs = self.model( 2025-12-04T09:47:16.9222832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9222905Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9223133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9223210Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9223448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9223522Z return func(*args, **kwargs) 2025-12-04T09:47:16.9223764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:47:16.9223871Z hidden_states = residual + hidden_states 2025-12-04T09:47:16.9223874Z 2025-12-04T09:47:16.9223977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9224172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9224241Z res = mod(**inputs) 2025-12-04T09:47:16.9224483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9224550Z outputs = self.model( 2025-12-04T09:47:16.9224801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9224870Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9225095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9225169Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9225408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9225486Z return func(*args, **kwargs) 2025-12-04T09:47:16.9225745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9225857Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9226109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9226179Z return func(*args, **kwargs) 2025-12-04T09:47:16.9226444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9226558Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9226562Z 2025-12-04T09:47:16.9226668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9226882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9226947Z res = mod(**inputs) 2025-12-04T09:47:16.9227232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9227303Z outputs = self.model( 2025-12-04T09:47:16.9227565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9227669Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9227903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9227987Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9228251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9228318Z return func(*args, **kwargs) 2025-12-04T09:47:16.9228573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9228691Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9228950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9229029Z return func(*args, **kwargs) 2025-12-04T09:47:16.9229289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:47:16.9229379Z key_states = self.k_proj(current_states) 2025-12-04T09:47:16.9229384Z 2025-12-04T09:47:16.9229491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9229697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9229768Z res = mod(**inputs) 2025-12-04T09:47:16.9230037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9230129Z outputs = self.model( 2025-12-04T09:47:16.9230399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9230473Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9230711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9230792Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9231043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9231120Z return func(*args, **kwargs) 2025-12-04T09:47:16.9231378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9231486Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9231739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9231810Z return func(*args, **kwargs) 2025-12-04T09:47:16.9232079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:47:16.9232192Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:47:16.9232199Z 2025-12-04T09:47:16.9232308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9232522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9232587Z res = mod(**inputs) 2025-12-04T09:47:16.9232854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9232924Z outputs = self.model( 2025-12-04T09:47:16.9233183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9233261Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9233500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9233580Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9233828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9233916Z return func(*args, **kwargs) 2025-12-04T09:47:16.9234169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9234271Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9234524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9234606Z return func(*args, **kwargs) 2025-12-04T09:47:16.9234873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:47:16.9235030Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:47:16.9235034Z 2025-12-04T09:47:16.9235137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9235336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9235406Z res = mod(**inputs) 2025-12-04T09:47:16.9235652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9235718Z outputs = self.model( 2025-12-04T09:47:16.9235971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9236039Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9236281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9236360Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9236605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9236680Z return func(*args, **kwargs) 2025-12-04T09:47:16.9236925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9237025Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9237267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9237337Z return func(*args, **kwargs) 2025-12-04T09:47:16.9237602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:47:16.9237694Z value_states = self.v_proj(current_states) 2025-12-04T09:47:16.9237698Z 2025-12-04T09:47:16.9237803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9238019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9238085Z res = mod(**inputs) 2025-12-04T09:47:16.9238351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9238422Z outputs = self.model( 2025-12-04T09:47:16.9238677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9238758Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9238990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9239070Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9239328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9239400Z return func(*args, **kwargs) 2025-12-04T09:47:16.9239684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9239789Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9240063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9240140Z return func(*args, **kwargs) 2025-12-04T09:47:16.9240399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:47:16.9240503Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:47:16.9240506Z 2025-12-04T09:47:16.9240613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9240820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9240893Z res = mod(**inputs) 2025-12-04T09:47:16.9241168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9241239Z outputs = self.model( 2025-12-04T09:47:16.9241505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9241581Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9241826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9241908Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9242166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9242243Z return func(*args, **kwargs) 2025-12-04T09:47:16.9242523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9242632Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9242886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9242958Z return func(*args, **kwargs) 2025-12-04T09:47:16.9243223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:47:16.9243356Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:47:16.9243360Z 2025-12-04T09:47:16.9243466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9243676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9243742Z res = mod(**inputs) 2025-12-04T09:47:16.9244009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9244077Z outputs = self.model( 2025-12-04T09:47:16.9244339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9244422Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9244658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9244740Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9245002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9245071Z return func(*args, **kwargs) 2025-12-04T09:47:16.9245335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:47:16.9245437Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:47:16.9245693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9245765Z return func(*args, **kwargs) 2025-12-04T09:47:16.9246032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:47:16.9246134Z attn_output = self.out_proj(attn_output) 2025-12-04T09:47:16.9246137Z 2025-12-04T09:47:16.9246237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9246423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9246488Z res = mod(**inputs) 2025-12-04T09:47:16.9246724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9246789Z outputs = self.model( 2025-12-04T09:47:16.9247040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9247106Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9247348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9247429Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9247670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9247743Z return func(*args, **kwargs) 2025-12-04T09:47:16.9247985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9248108Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9248112Z 2025-12-04T09:47:16.9248211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9248405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9248494Z res = mod(**inputs) 2025-12-04T09:47:16.9248742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9248807Z outputs = self.model( 2025-12-04T09:47:16.9249060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9249131Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9249359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9249437Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9249682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9249757Z return func(*args, **kwargs) 2025-12-04T09:47:16.9250003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:47:16.9250124Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:47:16.9250347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:47:16.9250415Z return self.act(input) 2025-12-04T09:47:16.9250419Z 2025-12-04T09:47:16.9250528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9250725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9250789Z res = mod(**inputs) 2025-12-04T09:47:16.9251049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:47:16.9251114Z outputs = self.model( 2025-12-04T09:47:16.9251364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:47:16.9251504Z layer_outputs = decoder_layer( 2025-12-04T09:47:16.9251742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:47:16.9251859Z return super().__call__(*args, **kwargs) 2025-12-04T09:47:16.9252126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:47:16.9252224Z return func(*args, **kwargs) 2025-12-04T09:47:16.9252518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:47:16.9252604Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:47:16.9252608Z 2025-12-04T09:47:16.9252726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9252936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9253005Z res = mod(**inputs) 2025-12-04T09:47:16.9253306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-12-04T09:47:16.9253386Z logits = self.lm_head(outputs[0]) 2025-12-04T09:47:16.9253392Z 2025-12-04T09:47:16.9253503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:47:16.9253693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:47:16.9253756Z res = mod(**inputs) 2025-12-04T09:47:16.9254011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 689, in forward 2025-12-04T09:47:16.9254083Z loss = self.loss_function( 2025-12-04T09:47:16.9254321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:47:16.9254504Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:47:16.9254773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:47:16.9254975Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:47:16.9254979Z 2025-12-04T09:47:29.4769688Z Compilation time (from dynamo_timed): 26.19412126 2025-12-04T09:47:29.4869429Z pass 2025-12-04T09:47:29.4869959Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:29.4876499Z TIMING: _recursive_pre_grad_passes:0.0126 _recursive_joint_graph_passes:0.77269 _recursive_post_grad_passes:0.25081 async_compile.wait:0.81427 code_gen:12.56803 inductor_compile:15.89868 backend_compile:21.43976 gc:0.00041 entire_frame_compile:26.19412 total_wall_time:26.19412 2025-12-04T09:47:29.4881338Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:17015 | FakeTensor.__torch_dispatch__:9977 | ProxyTorchDispatchMode.__torch_dispatch__:4882 2025-12-04T09:47:29.4883350Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-12-04T09:47:32.4644351Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:47:32.4645198Z import pynvml # type: ignore[import] 2025-12-04T09:47:35.8543425Z 2025-12-04T09:47:38.6435974Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:47:38.6442529Z loading model: 0it [00:02, ?it/s] 2025-12-04T09:47:38.6460738Z cpu eval XLNetLMHeadModel 2025-12-04T09:47:41.2817155Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:42.2368673Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:47:43.1954647Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:03.7167681Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7172775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7173256Z res = mod(**inputs) 2025-12-04T09:48:03.7173756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7174177Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7174591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-12-04T09:48:03.7175018Z word_emb_k = self.word_embedding(input_ids) 2025-12-04T09:48:03.7175176Z 2025-12-04T09:48:03.7175294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7175671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7176013Z res = mod(**inputs) 2025-12-04T09:48:03.7176432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7176842Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7177233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:48:03.7177686Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:48:03.7178184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:48:03.7178690Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:48:03.7179155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:48:03.7179711Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:48:03.7179923Z 2025-12-04T09:48:03.7180034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7180403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7180722Z res = mod(**inputs) 2025-12-04T09:48:03.7181089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7181490Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7181880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:48:03.7182319Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:48:03.7182827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:48:03.7183323Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:48:03.7183785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:48:03.7184284Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:48:03.7184492Z 2025-12-04T09:48:03.7184597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7184956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7185275Z res = mod(**inputs) 2025-12-04T09:48:03.7185645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7186057Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7186457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7186839Z outputs = layer_module( 2025-12-04T09:48:03.7187358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7187762Z outputs = self.rel_attn( 2025-12-04T09:48:03.7188153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7188549Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7188960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7189406Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7189575Z 2025-12-04T09:48:03.7189680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7190046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7190358Z res = mod(**inputs) 2025-12-04T09:48:03.7190721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7191106Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7191498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7192013Z outputs = layer_module( 2025-12-04T09:48:03.7192371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7192772Z outputs = self.rel_attn( 2025-12-04T09:48:03.7193144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7193546Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7193988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7194438Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7194610Z 2025-12-04T09:48:03.7194711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7195062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7195367Z res = mod(**inputs) 2025-12-04T09:48:03.7195717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7196130Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7196545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7196965Z outputs = layer_module( 2025-12-04T09:48:03.7197357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7197781Z outputs = self.rel_attn( 2025-12-04T09:48:03.7198146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7198536Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7198946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7199376Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7199536Z 2025-12-04T09:48:03.7199634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7199978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7200289Z res = mod(**inputs) 2025-12-04T09:48:03.7200648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7201035Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7201439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7201817Z outputs = layer_module( 2025-12-04T09:48:03.7202187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7202560Z outputs = self.rel_attn( 2025-12-04T09:48:03.7202917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7203309Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7203709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7204142Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7204302Z 2025-12-04T09:48:03.7204464Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7204818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7205137Z res = mod(**inputs) 2025-12-04T09:48:03.7205495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7205894Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7206277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7206655Z outputs = layer_module( 2025-12-04T09:48:03.7207026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7207428Z outputs = self.rel_attn( 2025-12-04T09:48:03.7207819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7208242Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7208688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7209158Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7209362Z 2025-12-04T09:48:03.7209475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7209863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7210203Z res = mod(**inputs) 2025-12-04T09:48:03.7210582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7210998Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7211417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7211929Z outputs = layer_module( 2025-12-04T09:48:03.7212315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7212733Z outputs = self.rel_attn( 2025-12-04T09:48:03.7213138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7213559Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7213993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7214460Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7214622Z 2025-12-04T09:48:03.7214733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7215081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7215402Z res = mod(**inputs) 2025-12-04T09:48:03.7215802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7216223Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7216656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7217067Z outputs = layer_module( 2025-12-04T09:48:03.7217468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7217854Z outputs = self.rel_attn( 2025-12-04T09:48:03.7218230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7218639Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7219118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7219553Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7219724Z 2025-12-04T09:48:03.7219827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7220184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7220503Z res = mod(**inputs) 2025-12-04T09:48:03.7220855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7221305Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7221697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7222091Z outputs = layer_module( 2025-12-04T09:48:03.7222473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7222855Z outputs = self.rel_attn( 2025-12-04T09:48:03.7223222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7223614Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7224032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7224559Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7224731Z 2025-12-04T09:48:03.7224856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7225204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7225524Z res = mod(**inputs) 2025-12-04T09:48:03.7225877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7226263Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7226659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7227040Z outputs = layer_module( 2025-12-04T09:48:03.7227410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7227786Z outputs = self.rel_attn( 2025-12-04T09:48:03.7228152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7228545Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7228949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7229387Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7229556Z 2025-12-04T09:48:03.7229659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7230028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7230360Z res = mod(**inputs) 2025-12-04T09:48:03.7230729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7231152Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7231553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7231936Z outputs = layer_module( 2025-12-04T09:48:03.7232304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7232694Z outputs = self.rel_attn( 2025-12-04T09:48:03.7233074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7233479Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7233895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7234339Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7234502Z 2025-12-04T09:48:03.7234603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7234955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7235275Z res = mod(**inputs) 2025-12-04T09:48:03.7235635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7236048Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7236445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7236849Z outputs = layer_module( 2025-12-04T09:48:03.7237229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7237638Z outputs = self.rel_attn( 2025-12-04T09:48:03.7238020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7238426Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7238833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7239271Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7239435Z 2025-12-04T09:48:03.7239554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7239925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7240254Z res = mod(**inputs) 2025-12-04T09:48:03.7240633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7241051Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7241456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7241862Z outputs = layer_module( 2025-12-04T09:48:03.7242248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7242649Z outputs = self.rel_attn( 2025-12-04T09:48:03.7243025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7243448Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7243906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7244370Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7244574Z 2025-12-04T09:48:03.7244701Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7245080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7245417Z res = mod(**inputs) 2025-12-04T09:48:03.7245790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7246212Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7246628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7247033Z outputs = layer_module( 2025-12-04T09:48:03.7247407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7247799Z outputs = self.rel_attn( 2025-12-04T09:48:03.7248174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7248572Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7248992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7249432Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7249596Z 2025-12-04T09:48:03.7249704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7250054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7250394Z res = mod(**inputs) 2025-12-04T09:48:03.7250751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7251148Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7251625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7252025Z outputs = layer_module( 2025-12-04T09:48:03.7252393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7252776Z outputs = self.rel_attn( 2025-12-04T09:48:03.7253159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7253589Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7254040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7254523Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7254700Z 2025-12-04T09:48:03.7254807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7255171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7255494Z res = mod(**inputs) 2025-12-04T09:48:03.7255847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7256245Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7256640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7257024Z outputs = layer_module( 2025-12-04T09:48:03.7257391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7257775Z outputs = self.rel_attn( 2025-12-04T09:48:03.7258164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7258558Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7258975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7259436Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7259597Z 2025-12-04T09:48:03.7259706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7260056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7260375Z res = mod(**inputs) 2025-12-04T09:48:03.7260734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7261126Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7261538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7261938Z outputs = layer_module( 2025-12-04T09:48:03.7262306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7262821Z outputs = self.rel_attn( 2025-12-04T09:48:03.7263201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7263600Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7264010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7264449Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7264649Z 2025-12-04T09:48:03.7264763Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7265114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7265425Z res = mod(**inputs) 2025-12-04T09:48:03.7265775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7266166Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7266555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7266917Z outputs = layer_module( 2025-12-04T09:48:03.7267274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7267648Z outputs = self.rel_attn( 2025-12-04T09:48:03.7267999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7268390Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7268802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7269229Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7269391Z 2025-12-04T09:48:03.7269491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7269840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7270155Z res = mod(**inputs) 2025-12-04T09:48:03.7270506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7270884Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7271276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7271660Z outputs = layer_module( 2025-12-04T09:48:03.7272024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7272606Z outputs = self.rel_attn( 2025-12-04T09:48:03.7272974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7273435Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7273834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7274266Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7274425Z 2025-12-04T09:48:03.7274535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7274878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7275185Z res = mod(**inputs) 2025-12-04T09:48:03.7275563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7275973Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7276359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7276739Z outputs = layer_module( 2025-12-04T09:48:03.7277097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7277465Z outputs = self.rel_attn( 2025-12-04T09:48:03.7277818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7278206Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7278647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7279103Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7279272Z 2025-12-04T09:48:03.7279380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7279737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7280069Z res = mod(**inputs) 2025-12-04T09:48:03.7280510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7280902Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7281294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7281661Z outputs = layer_module( 2025-12-04T09:48:03.7282013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7282386Z outputs = self.rel_attn( 2025-12-04T09:48:03.7282758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7283155Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7283564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7284000Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7284172Z 2025-12-04T09:48:03.7284276Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7284629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7284949Z res = mod(**inputs) 2025-12-04T09:48:03.7285319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7285710Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7286136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7286512Z outputs = layer_module( 2025-12-04T09:48:03.7287670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7288225Z outputs = self.rel_attn( 2025-12-04T09:48:03.7288599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7289005Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7289423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7289874Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7290037Z 2025-12-04T09:48:03.7290143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7290530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7290856Z res = mod(**inputs) 2025-12-04T09:48:03.7291218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7291670Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7292094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7292501Z outputs = layer_module( 2025-12-04T09:48:03.7292883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7293280Z outputs = self.rel_attn( 2025-12-04T09:48:03.7293668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7294065Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7294456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7294877Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7295030Z 2025-12-04T09:48:03.7295134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7295459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7295818Z res = mod(**inputs) 2025-12-04T09:48:03.7296227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7296601Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7296964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7297324Z outputs = layer_module( 2025-12-04T09:48:03.7297674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7298035Z outputs = self.rel_attn( 2025-12-04T09:48:03.7298375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7298758Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7299150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7299562Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7299727Z 2025-12-04T09:48:03.7299825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7300172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7300483Z res = mod(**inputs) 2025-12-04T09:48:03.7300838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7301213Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7301603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7301961Z outputs = layer_module( 2025-12-04T09:48:03.7302297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7302656Z outputs = self.rel_attn( 2025-12-04T09:48:03.7302999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7303364Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7303773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7304195Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7304348Z 2025-12-04T09:48:03.7304451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7304785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7305086Z res = mod(**inputs) 2025-12-04T09:48:03.7305425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7305792Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7306163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7306568Z outputs = layer_module( 2025-12-04T09:48:03.7306931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7307286Z outputs = self.rel_attn( 2025-12-04T09:48:03.7307632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7308019Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7308162Z 2025-12-04T09:48:03.7308264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7308587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7308886Z res = mod(**inputs) 2025-12-04T09:48:03.7309222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7309586Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7309954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7310314Z outputs = layer_module( 2025-12-04T09:48:03.7310656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7311017Z outputs = self.rel_attn( 2025-12-04T09:48:03.7311350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7311728Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7311868Z 2025-12-04T09:48:03.7311961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7312285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7312580Z res = mod(**inputs) 2025-12-04T09:48:03.7312907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7313269Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7313653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7314075Z outputs = layer_module( 2025-12-04T09:48:03.7314410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7314770Z outputs = self.rel_attn( 2025-12-04T09:48:03.7315109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7315472Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7315835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7316267Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7316448Z 2025-12-04T09:48:03.7316545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7316892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7317195Z res = mod(**inputs) 2025-12-04T09:48:03.7317550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7317928Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7318302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:48:03.7318732Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:48:03.7319203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:48:03.7319681Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:48:03.7320156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:48:03.7320637Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:48:03.7320835Z 2025-12-04T09:48:03.7320933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7321267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7321563Z res = mod(**inputs) 2025-12-04T09:48:03.7321901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7322273Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7322653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7323001Z outputs = layer_module( 2025-12-04T09:48:03.7323339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7323691Z outputs = self.rel_attn( 2025-12-04T09:48:03.7324022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7324434Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7324613Z 2025-12-04T09:48:03.7324708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7325030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7325316Z res = mod(**inputs) 2025-12-04T09:48:03.7325645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7326011Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7326374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7326721Z outputs = layer_module( 2025-12-04T09:48:03.7327082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7327446Z outputs = self.rel_attn( 2025-12-04T09:48:03.7327776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7328128Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7328494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7328914Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7329081Z 2025-12-04T09:48:03.7329176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7329498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7329812Z res = mod(**inputs) 2025-12-04T09:48:03.7330151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7330523Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7330897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7331257Z outputs = layer_module( 2025-12-04T09:48:03.7331668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7332051Z outputs = self.rel_attn( 2025-12-04T09:48:03.7332426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7332862Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7333017Z 2025-12-04T09:48:03.7333122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7333480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7333799Z res = mod(**inputs) 2025-12-04T09:48:03.7334160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7334567Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7334939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7335318Z outputs = layer_module( 2025-12-04T09:48:03.7335662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7336015Z outputs = self.rel_attn( 2025-12-04T09:48:03.7336359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7336724Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7337104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7337531Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7337703Z 2025-12-04T09:48:03.7337800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7338133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7338432Z res = mod(**inputs) 2025-12-04T09:48:03.7338764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7339134Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7339506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7339862Z outputs = layer_module( 2025-12-04T09:48:03.7340228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7340597Z outputs = self.rel_attn( 2025-12-04T09:48:03.7340969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7341350Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7341757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7342200Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7342356Z 2025-12-04T09:48:03.7342458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7342788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7343107Z res = mod(**inputs) 2025-12-04T09:48:03.7343446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7343812Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7344185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7344550Z outputs = layer_module( 2025-12-04T09:48:03.7344897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7345250Z outputs = self.rel_attn( 2025-12-04T09:48:03.7345596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7345993Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7346383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7346806Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7346968Z 2025-12-04T09:48:03.7347066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7347403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7347710Z res = mod(**inputs) 2025-12-04T09:48:03.7348041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7348406Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7348769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7349115Z outputs = layer_module( 2025-12-04T09:48:03.7349449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7349931Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7350408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7350776Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7351132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7351488Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7351828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7352181Z output = self.layer_1(output) 2025-12-04T09:48:03.7352298Z 2025-12-04T09:48:03.7352401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7352732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7353070Z res = mod(**inputs) 2025-12-04T09:48:03.7353409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7353791Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7354147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7354500Z outputs = layer_module( 2025-12-04T09:48:03.7354839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7355319Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7355805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7356192Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7356558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7356921Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7357264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7357637Z output = self.activation_function(output) 2025-12-04T09:48:03.7357967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7358278Z return self.act(input) 2025-12-04T09:48:03.7358386Z 2025-12-04T09:48:03.7358481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7358806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7359118Z res = mod(**inputs) 2025-12-04T09:48:03.7359447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7359815Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7360179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7360526Z outputs = layer_module( 2025-12-04T09:48:03.7360863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7361338Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7361823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7362191Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7362548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7362909Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7363257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7363610Z output = self.layer_2(output) 2025-12-04T09:48:03.7363731Z 2025-12-04T09:48:03.7363828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7364157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7364444Z res = mod(**inputs) 2025-12-04T09:48:03.7364781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7365151Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7365516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7365864Z outputs = layer_module( 2025-12-04T09:48:03.7366226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7366599Z outputs = self.rel_attn( 2025-12-04T09:48:03.7366944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7367313Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7367459Z 2025-12-04T09:48:03.7367555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7367882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7368170Z res = mod(**inputs) 2025-12-04T09:48:03.7368504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7368887Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7369253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7369598Z outputs = layer_module( 2025-12-04T09:48:03.7369945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7370307Z outputs = self.rel_attn( 2025-12-04T09:48:03.7370650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7371039Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7371199Z 2025-12-04T09:48:03.7371296Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7371748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7372143Z res = mod(**inputs) 2025-12-04T09:48:03.7372727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7373169Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7373583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7373963Z outputs = layer_module( 2025-12-04T09:48:03.7374332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7374724Z outputs = self.rel_attn( 2025-12-04T09:48:03.7375068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7375437Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7375820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7376257Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7376433Z 2025-12-04T09:48:03.7376532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7376867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7377171Z res = mod(**inputs) 2025-12-04T09:48:03.7377504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7377875Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7378245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7378612Z outputs = layer_module( 2025-12-04T09:48:03.7378951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7379314Z outputs = self.rel_attn( 2025-12-04T09:48:03.7379712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7380141Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7380348Z 2025-12-04T09:48:03.7380446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7380786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7381092Z res = mod(**inputs) 2025-12-04T09:48:03.7381427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7381807Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7382185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7382557Z outputs = layer_module( 2025-12-04T09:48:03.7382926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7383295Z outputs = self.rel_attn( 2025-12-04T09:48:03.7383650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7384023Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7384396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7384839Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7385012Z 2025-12-04T09:48:03.7385118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7385450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7385790Z res = mod(**inputs) 2025-12-04T09:48:03.7386125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7386499Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7386869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7387240Z outputs = layer_module( 2025-12-04T09:48:03.7387591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7387950Z outputs = self.rel_attn( 2025-12-04T09:48:03.7388307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7388691Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7388832Z 2025-12-04T09:48:03.7388934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7389266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7389567Z res = mod(**inputs) 2025-12-04T09:48:03.7389897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7390271Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7390632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7390995Z outputs = layer_module( 2025-12-04T09:48:03.7391341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7391697Z outputs = self.rel_attn( 2025-12-04T09:48:03.7392043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7392410Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7392814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7393240Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7393428Z 2025-12-04T09:48:03.7393526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7393859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7394155Z res = mod(**inputs) 2025-12-04T09:48:03.7394495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7394873Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7395246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7395606Z outputs = layer_module( 2025-12-04T09:48:03.7395976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7396344Z outputs = self.rel_attn( 2025-12-04T09:48:03.7396691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7397070Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7397477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7397891Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7398042Z 2025-12-04T09:48:03.7398139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7398470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7398787Z res = mod(**inputs) 2025-12-04T09:48:03.7399125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7399497Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7399861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7400218Z outputs = layer_module( 2025-12-04T09:48:03.7400557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7400902Z outputs = self.rel_attn( 2025-12-04T09:48:03.7401254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7401630Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7402023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7402453Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7402618Z 2025-12-04T09:48:03.7402714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7403058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7403352Z res = mod(**inputs) 2025-12-04T09:48:03.7403682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7404048Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7404408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7404760Z outputs = layer_module( 2025-12-04T09:48:03.7405099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7407817Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7408350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7408762Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7409138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7409503Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7409867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7410238Z output = self.layer_1(output) 2025-12-04T09:48:03.7410355Z 2025-12-04T09:48:03.7410464Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7410803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7411141Z res = mod(**inputs) 2025-12-04T09:48:03.7411611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7412005Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7412396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7412772Z outputs = layer_module( 2025-12-04T09:48:03.7413123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7413645Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7414203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7414651Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7415052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7415462Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7415828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7416217Z output = self.activation_function(output) 2025-12-04T09:48:03.7416560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7416895Z return self.act(input) 2025-12-04T09:48:03.7417002Z 2025-12-04T09:48:03.7417112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7417452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7417763Z res = mod(**inputs) 2025-12-04T09:48:03.7418115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7418492Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7418858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7419224Z outputs = layer_module( 2025-12-04T09:48:03.7419570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7420058Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7420544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7420922Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7421287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7421646Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7422068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7422495Z output = self.layer_2(output) 2025-12-04T09:48:03.7422609Z 2025-12-04T09:48:03.7422714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7423047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7423353Z res = mod(**inputs) 2025-12-04T09:48:03.7423694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7424075Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7424444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7424808Z outputs = layer_module( 2025-12-04T09:48:03.7425187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7425549Z outputs = self.rel_attn( 2025-12-04T09:48:03.7425897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7426288Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7426432Z 2025-12-04T09:48:03.7426537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7426866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7427164Z res = mod(**inputs) 2025-12-04T09:48:03.7427498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7427888Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7428262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7428625Z outputs = layer_module( 2025-12-04T09:48:03.7428970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7429323Z outputs = self.rel_attn( 2025-12-04T09:48:03.7429670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7430057Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7430200Z 2025-12-04T09:48:03.7430305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7430635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7430936Z res = mod(**inputs) 2025-12-04T09:48:03.7431277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7431647Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7432018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7432379Z outputs = layer_module( 2025-12-04T09:48:03.7432730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7433083Z outputs = self.rel_attn( 2025-12-04T09:48:03.7433433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7433802Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7434178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7434615Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7434801Z 2025-12-04T09:48:03.7434937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7435276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7435589Z res = mod(**inputs) 2025-12-04T09:48:03.7435930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7436308Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7436683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7437043Z outputs = layer_module( 2025-12-04T09:48:03.7437391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7437756Z outputs = self.rel_attn( 2025-12-04T09:48:03.7438118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7438546Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7438726Z 2025-12-04T09:48:03.7438827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7439159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7439452Z res = mod(**inputs) 2025-12-04T09:48:03.7439792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7440171Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7440546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7440920Z outputs = layer_module( 2025-12-04T09:48:03.7441269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7441629Z outputs = self.rel_attn( 2025-12-04T09:48:03.7441969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7442339Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7442721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7443156Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7443329Z 2025-12-04T09:48:03.7443426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7443764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7444064Z res = mod(**inputs) 2025-12-04T09:48:03.7444395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7444775Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7445143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7445506Z outputs = layer_module( 2025-12-04T09:48:03.7445844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7446203Z outputs = self.rel_attn( 2025-12-04T09:48:03.7446547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7446937Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7447079Z 2025-12-04T09:48:03.7447180Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7447517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7447823Z res = mod(**inputs) 2025-12-04T09:48:03.7448192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7448597Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7448979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7449350Z outputs = layer_module( 2025-12-04T09:48:03.7449699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7450067Z outputs = self.rel_attn( 2025-12-04T09:48:03.7450425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7450796Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7451207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7451766Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7451960Z 2025-12-04T09:48:03.7452079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7452474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7452811Z res = mod(**inputs) 2025-12-04T09:48:03.7453200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7453592Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7453975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7454375Z outputs = layer_module( 2025-12-04T09:48:03.7454730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7455096Z outputs = self.rel_attn( 2025-12-04T09:48:03.7455454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7455847Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7456260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7456685Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7456851Z 2025-12-04T09:48:03.7456951Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7457293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7457619Z res = mod(**inputs) 2025-12-04T09:48:03.7457965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7458354Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7458738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7459101Z outputs = layer_module( 2025-12-04T09:48:03.7459456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7459821Z outputs = self.rel_attn( 2025-12-04T09:48:03.7460172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7460556Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7460958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7461392Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7461554Z 2025-12-04T09:48:03.7461691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7462034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7462363Z res = mod(**inputs) 2025-12-04T09:48:03.7462713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7463097Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7463486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7463857Z outputs = layer_module( 2025-12-04T09:48:03.7464212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7464714Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7465248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7465634Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7466001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7466371Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7466731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7467097Z output = self.layer_1(output) 2025-12-04T09:48:03.7467216Z 2025-12-04T09:48:03.7467320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7467658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7467984Z res = mod(**inputs) 2025-12-04T09:48:03.7468327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7468701Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7469077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7469444Z outputs = layer_module( 2025-12-04T09:48:03.7469784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7470275Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7470772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7471148Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7471517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7471891Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7472482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7472892Z output = self.activation_function(output) 2025-12-04T09:48:03.7473234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7473569Z return self.act(input) 2025-12-04T09:48:03.7473674Z 2025-12-04T09:48:03.7473782Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7474116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7474429Z res = mod(**inputs) 2025-12-04T09:48:03.7474780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7475165Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7475609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7475974Z outputs = layer_module( 2025-12-04T09:48:03.7476362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7476850Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7477349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7477729Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7478099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7478463Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7478849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7479223Z output = self.layer_2(output) 2025-12-04T09:48:03.7479342Z 2025-12-04T09:48:03.7479451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7479790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7480101Z res = mod(**inputs) 2025-12-04T09:48:03.7480966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7481356Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7481726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7482123Z outputs = layer_module( 2025-12-04T09:48:03.7482470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7482832Z outputs = self.rel_attn( 2025-12-04T09:48:03.7483187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7483582Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7483730Z 2025-12-04T09:48:03.7483838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7484174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7484486Z res = mod(**inputs) 2025-12-04T09:48:03.7484831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7485207Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7485584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7485951Z outputs = layer_module( 2025-12-04T09:48:03.7486310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7486680Z outputs = self.rel_attn( 2025-12-04T09:48:03.7487049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7487467Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7487616Z 2025-12-04T09:48:03.7487724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7488063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7488372Z res = mod(**inputs) 2025-12-04T09:48:03.7488723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7489102Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7489506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7489871Z outputs = layer_module( 2025-12-04T09:48:03.7490240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7490610Z outputs = self.rel_attn( 2025-12-04T09:48:03.7490976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7491374Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7491868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7492390Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7492596Z 2025-12-04T09:48:03.7492726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7493118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7493432Z res = mod(**inputs) 2025-12-04T09:48:03.7493796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7494194Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7494593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7494969Z outputs = layer_module( 2025-12-04T09:48:03.7495332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7495726Z outputs = self.rel_attn( 2025-12-04T09:48:03.7496082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7496526Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7496716Z 2025-12-04T09:48:03.7496816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7497174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7497493Z res = mod(**inputs) 2025-12-04T09:48:03.7497840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7498229Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7498606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7498979Z outputs = layer_module( 2025-12-04T09:48:03.7499340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7499714Z outputs = self.rel_attn( 2025-12-04T09:48:03.7500067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7500448Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7500843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7501292Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7501470Z 2025-12-04T09:48:03.7501569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7501921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7502240Z res = mod(**inputs) 2025-12-04T09:48:03.7502584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7502976Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7503386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7503765Z outputs = layer_module( 2025-12-04T09:48:03.7504134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7504505Z outputs = self.rel_attn( 2025-12-04T09:48:03.7504862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7505261Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7505412Z 2025-12-04T09:48:03.7505512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7505852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7506160Z res = mod(**inputs) 2025-12-04T09:48:03.7506519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7506916Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7507306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7507678Z outputs = layer_module( 2025-12-04T09:48:03.7508026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7508403Z outputs = self.rel_attn( 2025-12-04T09:48:03.7508762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7509133Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7509545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7509986Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7510154Z 2025-12-04T09:48:03.7510260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7510597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7510903Z res = mod(**inputs) 2025-12-04T09:48:03.7511250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7511637Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7512006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7512375Z outputs = layer_module( 2025-12-04T09:48:03.7512731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7513114Z outputs = self.rel_attn( 2025-12-04T09:48:03.7513476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7513871Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7514302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7514723Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7514885Z 2025-12-04T09:48:03.7514982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7515325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7515632Z res = mod(**inputs) 2025-12-04T09:48:03.7515972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7516368Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7516780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7517152Z outputs = layer_module( 2025-12-04T09:48:03.7517544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7517910Z outputs = self.rel_attn( 2025-12-04T09:48:03.7518267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7518650Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7519065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7519513Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7519677Z 2025-12-04T09:48:03.7519798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7520160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7520490Z res = mod(**inputs) 2025-12-04T09:48:03.7520838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7521217Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7521609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7521983Z outputs = layer_module( 2025-12-04T09:48:03.7522345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7522831Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7523356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7523734Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7524096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7524465Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7524828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7525202Z output = self.layer_1(output) 2025-12-04T09:48:03.7525318Z 2025-12-04T09:48:03.7525419Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7525766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7526081Z res = mod(**inputs) 2025-12-04T09:48:03.7526427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7526807Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7527193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7527567Z outputs = layer_module( 2025-12-04T09:48:03.7527912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7528415Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7528921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7529311Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7529677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7530054Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7530447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7530839Z output = self.activation_function(output) 2025-12-04T09:48:03.7531205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7531629Z return self.act(input) 2025-12-04T09:48:03.7531744Z 2025-12-04T09:48:03.7531858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7532210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7532544Z res = mod(**inputs) 2025-12-04T09:48:03.7532926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7533355Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7533782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7534160Z outputs = layer_module( 2025-12-04T09:48:03.7534525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7535035Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7535563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7535960Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7536348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7536749Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7537121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7537498Z output = self.layer_2(output) 2025-12-04T09:48:03.7537616Z 2025-12-04T09:48:03.7537725Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7538071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7538380Z res = mod(**inputs) 2025-12-04T09:48:03.7538727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7539106Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7539489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7539856Z outputs = layer_module( 2025-12-04T09:48:03.7540225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7540592Z outputs = self.rel_attn( 2025-12-04T09:48:03.7540956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7541357Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7541507Z 2025-12-04T09:48:03.7541618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7541958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7542269Z res = mod(**inputs) 2025-12-04T09:48:03.7542622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7543008Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7543401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7543776Z outputs = layer_module( 2025-12-04T09:48:03.7544163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7544522Z outputs = self.rel_attn( 2025-12-04T09:48:03.7544887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7545279Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7545422Z 2025-12-04T09:48:03.7545519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7545851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7546148Z res = mod(**inputs) 2025-12-04T09:48:03.7546486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7546857Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7547247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7547614Z outputs = layer_module( 2025-12-04T09:48:03.7547955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7548322Z outputs = self.rel_attn( 2025-12-04T09:48:03.7548676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7549045Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7549416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7549856Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7550056Z 2025-12-04T09:48:03.7550156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7550505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7550805Z res = mod(**inputs) 2025-12-04T09:48:03.7551151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7551539Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7551912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7552286Z outputs = layer_module( 2025-12-04T09:48:03.7552639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7553011Z outputs = self.rel_attn( 2025-12-04T09:48:03.7553360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7553789Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7553971Z 2025-12-04T09:48:03.7554081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7554425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7554727Z res = mod(**inputs) 2025-12-04T09:48:03.7555074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7555472Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7555845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7556222Z outputs = layer_module( 2025-12-04T09:48:03.7556575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7556949Z outputs = self.rel_attn( 2025-12-04T09:48:03.7557322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7557697Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7558083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7558529Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7558708Z 2025-12-04T09:48:03.7558806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7559144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7559440Z res = mod(**inputs) 2025-12-04T09:48:03.7559765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7560143Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7560531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7560897Z outputs = layer_module( 2025-12-04T09:48:03.7561237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7561607Z outputs = self.rel_attn( 2025-12-04T09:48:03.7561954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7562338Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7562490Z 2025-12-04T09:48:03.7562588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7562924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7563250Z res = mod(**inputs) 2025-12-04T09:48:03.7563589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7563970Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7564343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7564710Z outputs = layer_module( 2025-12-04T09:48:03.7565049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7565422Z outputs = self.rel_attn( 2025-12-04T09:48:03.7565781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7566147Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7566540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7566985Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7567154Z 2025-12-04T09:48:03.7567267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7567607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7567924Z res = mod(**inputs) 2025-12-04T09:48:03.7568275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7568662Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7569052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7569428Z outputs = layer_module( 2025-12-04T09:48:03.7569787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7570157Z outputs = self.rel_attn( 2025-12-04T09:48:03.7570542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7570941Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7571359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7571884Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7572069Z 2025-12-04T09:48:03.7572179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7572728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7573066Z res = mod(**inputs) 2025-12-04T09:48:03.7573449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7573888Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7574319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7574690Z outputs = layer_module( 2025-12-04T09:48:03.7575044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7575424Z outputs = self.rel_attn( 2025-12-04T09:48:03.7575774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7576168Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7576579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7577016Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7577217Z 2025-12-04T09:48:03.7577319Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7577673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7577984Z res = mod(**inputs) 2025-12-04T09:48:03.7578333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7578717Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7579106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7579482Z outputs = layer_module( 2025-12-04T09:48:03.7579835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7580344Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7580862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7581253Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7581628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7582013Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7582381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7582757Z output = self.layer_1(output) 2025-12-04T09:48:03.7582876Z 2025-12-04T09:48:03.7582975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7583317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7583628Z res = mod(**inputs) 2025-12-04T09:48:03.7583968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7584362Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7584775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7585150Z outputs = layer_module( 2025-12-04T09:48:03.7585522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7586025Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7586543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7586926Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7587294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7587672Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7588058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7588443Z output = self.activation_function(output) 2025-12-04T09:48:03.7588783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7589113Z return self.act(input) 2025-12-04T09:48:03.7589218Z 2025-12-04T09:48:03.7589322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7589662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7589974Z res = mod(**inputs) 2025-12-04T09:48:03.7590318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7590722Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7591098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7591470Z outputs = layer_module( 2025-12-04T09:48:03.7591825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7592325Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7592827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7593200Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7593567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7593928Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7594285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7594646Z output = self.layer_2(output) 2025-12-04T09:48:03.7594759Z 2025-12-04T09:48:03.7594863Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7595195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7595497Z res = mod(**inputs) 2025-12-04T09:48:03.7595833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7596202Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7596575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7596935Z outputs = layer_module( 2025-12-04T09:48:03.7597278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7597632Z outputs = self.rel_attn( 2025-12-04T09:48:03.7598002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7598394Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7598556Z 2025-12-04T09:48:03.7598653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7598997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7599298Z res = mod(**inputs) 2025-12-04T09:48:03.7599635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7600012Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7600383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7600750Z outputs = layer_module( 2025-12-04T09:48:03.7601111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7601479Z outputs = self.rel_attn( 2025-12-04T09:48:03.7601832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7602225Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7602367Z 2025-12-04T09:48:03.7602464Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7602799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7603102Z res = mod(**inputs) 2025-12-04T09:48:03.7603440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7603824Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7604196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7604568Z outputs = layer_module( 2025-12-04T09:48:03.7604913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7605285Z outputs = self.rel_attn( 2025-12-04T09:48:03.7605644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7606018Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7606401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7606850Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7607031Z 2025-12-04T09:48:03.7607147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7607504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7607817Z res = mod(**inputs) 2025-12-04T09:48:03.7608177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7608588Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7608972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7609356Z outputs = layer_module( 2025-12-04T09:48:03.7609720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7610097Z outputs = self.rel_attn( 2025-12-04T09:48:03.7610455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7610908Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7611094Z 2025-12-04T09:48:03.7611224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7611642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7612004Z res = mod(**inputs) 2025-12-04T09:48:03.7612388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7612811Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7613197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7613582Z outputs = layer_module( 2025-12-04T09:48:03.7613955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7614330Z outputs = self.rel_attn( 2025-12-04T09:48:03.7614694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7615075Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7615461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7615902Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7616085Z 2025-12-04T09:48:03.7616186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7616527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7616833Z res = mod(**inputs) 2025-12-04T09:48:03.7617170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7617581Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7617961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7618333Z outputs = layer_module( 2025-12-04T09:48:03.7618735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7619107Z outputs = self.rel_attn( 2025-12-04T09:48:03.7619460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7619855Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7620009Z 2025-12-04T09:48:03.7620108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7620451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7620760Z res = mod(**inputs) 2025-12-04T09:48:03.7621101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7621496Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7621874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7622242Z outputs = layer_module( 2025-12-04T09:48:03.7622598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7622968Z outputs = self.rel_attn( 2025-12-04T09:48:03.7623379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7623734Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7624112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7624542Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7624706Z 2025-12-04T09:48:03.7624811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7625161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7625462Z res = mod(**inputs) 2025-12-04T09:48:03.7625824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7626193Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7626567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7626926Z outputs = layer_module( 2025-12-04T09:48:03.7627271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7627627Z outputs = self.rel_attn( 2025-12-04T09:48:03.7628004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7628391Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7628786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7629213Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7629376Z 2025-12-04T09:48:03.7629473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7629812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7630107Z res = mod(**inputs) 2025-12-04T09:48:03.7630445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7630838Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7631207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7631560Z outputs = layer_module( 2025-12-04T09:48:03.7631920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7632283Z outputs = self.rel_attn( 2025-12-04T09:48:03.7632619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7632998Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7633390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7633810Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7633965Z 2025-12-04T09:48:03.7634065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7634399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7634700Z res = mod(**inputs) 2025-12-04T09:48:03.7635031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7635408Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7635777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7636139Z outputs = layer_module( 2025-12-04T09:48:03.7636478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7636969Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7637463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7637837Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7638263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7638641Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7639020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7639387Z output = self.layer_1(output) 2025-12-04T09:48:03.7639503Z 2025-12-04T09:48:03.7639602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7639943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7640246Z res = mod(**inputs) 2025-12-04T09:48:03.7640576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7640955Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7641347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7641719Z outputs = layer_module( 2025-12-04T09:48:03.7642080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7642593Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7643111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7643502Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7643880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7645205Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7645575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7645957Z output = self.activation_function(output) 2025-12-04T09:48:03.7646308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7646646Z return self.act(input) 2025-12-04T09:48:03.7646753Z 2025-12-04T09:48:03.7646860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7647197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7647507Z res = mod(**inputs) 2025-12-04T09:48:03.7647855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7648236Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7648618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7648992Z outputs = layer_module( 2025-12-04T09:48:03.7649356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7649853Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7650358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7650739Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7651114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7651654Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7652062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7652488Z output = self.layer_2(output) 2025-12-04T09:48:03.7652617Z 2025-12-04T09:48:03.7652741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7653120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7653437Z res = mod(**inputs) 2025-12-04T09:48:03.7653802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7653881Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7654126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7654201Z outputs = layer_module( 2025-12-04T09:48:03.7654445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7654516Z outputs = self.rel_attn( 2025-12-04T09:48:03.7654780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7654879Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7654883Z 2025-12-04T09:48:03.7654992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7655185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7655247Z res = mod(**inputs) 2025-12-04T09:48:03.7655504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7655584Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7655839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7655920Z outputs = layer_module( 2025-12-04T09:48:03.7656165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7656242Z outputs = self.rel_attn( 2025-12-04T09:48:03.7656492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7656591Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7656601Z 2025-12-04T09:48:03.7656703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7656894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7656961Z res = mod(**inputs) 2025-12-04T09:48:03.7657207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7657284Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7657538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7657604Z outputs = layer_module( 2025-12-04T09:48:03.7657854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7657920Z outputs = self.rel_attn( 2025-12-04T09:48:03.7658167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7658246Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7658509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7658634Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7658647Z 2025-12-04T09:48:03.7658745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7658934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7659003Z res = mod(**inputs) 2025-12-04T09:48:03.7659272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7659351Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7659619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7659683Z outputs = layer_module( 2025-12-04T09:48:03.7659935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7659999Z outputs = self.rel_attn( 2025-12-04T09:48:03.7660246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7660384Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7660388Z 2025-12-04T09:48:03.7660485Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7660692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7660762Z res = mod(**inputs) 2025-12-04T09:48:03.7661018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7661105Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7661349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7661413Z outputs = layer_module( 2025-12-04T09:48:03.7661665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7661730Z outputs = self.rel_attn( 2025-12-04T09:48:03.7662000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7662076Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7662330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7662456Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7662461Z 2025-12-04T09:48:03.7662556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7662742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7662809Z res = mod(**inputs) 2025-12-04T09:48:03.7663047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7663127Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7663367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7663430Z outputs = layer_module( 2025-12-04T09:48:03.7663674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7663737Z outputs = self.rel_attn( 2025-12-04T09:48:03.7663973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7664076Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7664080Z 2025-12-04T09:48:03.7664174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7664366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7664425Z res = mod(**inputs) 2025-12-04T09:48:03.7664664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7664748Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7665007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7665079Z outputs = layer_module( 2025-12-04T09:48:03.7665321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7665409Z outputs = self.rel_attn( 2025-12-04T09:48:03.7665656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7665721Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7665977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7666100Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7666105Z 2025-12-04T09:48:03.7666199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7666406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7666466Z res = mod(**inputs) 2025-12-04T09:48:03.7666706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7666789Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7667029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7667098Z outputs = layer_module( 2025-12-04T09:48:03.7667340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7667402Z outputs = self.rel_attn( 2025-12-04T09:48:03.7667661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7667748Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7668007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7668119Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7668124Z 2025-12-04T09:48:03.7668218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7668408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7668467Z res = mod(**inputs) 2025-12-04T09:48:03.7668702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7668786Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7669026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7669098Z outputs = layer_module( 2025-12-04T09:48:03.7669334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7669398Z outputs = self.rel_attn( 2025-12-04T09:48:03.7669639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7669723Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7669980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7670090Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7670093Z 2025-12-04T09:48:03.7670187Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7670379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7670439Z res = mod(**inputs) 2025-12-04T09:48:03.7670692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7670780Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7671023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7671109Z outputs = layer_module( 2025-12-04T09:48:03.7671346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7671539Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7671791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7671865Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7672122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7672199Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7672613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7672698Z output = self.layer_1(output) 2025-12-04T09:48:03.7672702Z 2025-12-04T09:48:03.7672799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7672985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7673055Z res = mod(**inputs) 2025-12-04T09:48:03.7673302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7673439Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7673680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7673745Z outputs = layer_module( 2025-12-04T09:48:03.7673991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7674186Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7674432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7674521Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7674761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7674836Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7675075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7675157Z output = self.activation_function(output) 2025-12-04T09:48:03.7675376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7675444Z return self.act(input) 2025-12-04T09:48:03.7675449Z 2025-12-04T09:48:03.7675553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7675739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7675797Z res = mod(**inputs) 2025-12-04T09:48:03.7676043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7676119Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7676357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7676430Z outputs = layer_module( 2025-12-04T09:48:03.7676695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7676900Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7677164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7677236Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7677482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7677550Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7677794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7677862Z output = self.layer_2(output) 2025-12-04T09:48:03.7677866Z 2025-12-04T09:48:03.7677964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7678181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7678243Z res = mod(**inputs) 2025-12-04T09:48:03.7678488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7678577Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7678816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7678885Z outputs = layer_module( 2025-12-04T09:48:03.7679126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7679190Z outputs = self.rel_attn( 2025-12-04T09:48:03.7679452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7679544Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7679549Z 2025-12-04T09:48:03.7679650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7679831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7679892Z res = mod(**inputs) 2025-12-04T09:48:03.7680146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7680218Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7680450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7680517Z outputs = layer_module( 2025-12-04T09:48:03.7680747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7680816Z outputs = self.rel_attn( 2025-12-04T09:48:03.7681048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7681140Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7681144Z 2025-12-04T09:48:03.7681243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7681419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7681474Z res = mod(**inputs) 2025-12-04T09:48:03.7681712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7681786Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7682026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7682088Z outputs = layer_module( 2025-12-04T09:48:03.7682336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7682411Z outputs = self.rel_attn( 2025-12-04T09:48:03.7682652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7682745Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7683008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7683130Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7683135Z 2025-12-04T09:48:03.7683238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7683424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7683484Z res = mod(**inputs) 2025-12-04T09:48:03.7683757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7683837Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7684081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7684148Z outputs = layer_module( 2025-12-04T09:48:03.7684385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7684457Z outputs = self.rel_attn( 2025-12-04T09:48:03.7684694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7684823Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7684843Z 2025-12-04T09:48:03.7684942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7685129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7685198Z res = mod(**inputs) 2025-12-04T09:48:03.7685439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7685518Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7685768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7685832Z outputs = layer_module( 2025-12-04T09:48:03.7686080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7686144Z outputs = self.rel_attn( 2025-12-04T09:48:03.7686382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7686458Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7686719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7686847Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7686852Z 2025-12-04T09:48:03.7686950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7687133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7687201Z res = mod(**inputs) 2025-12-04T09:48:03.7687443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7687519Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7687769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7687834Z outputs = layer_module( 2025-12-04T09:48:03.7688109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7688175Z outputs = self.rel_attn( 2025-12-04T09:48:03.7688412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7688530Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7688534Z 2025-12-04T09:48:03.7688627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7688819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7688878Z res = mod(**inputs) 2025-12-04T09:48:03.7689123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7689208Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7689468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7689537Z outputs = layer_module( 2025-12-04T09:48:03.7689787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7689853Z outputs = self.rel_attn( 2025-12-04T09:48:03.7690102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7690169Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7690434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7690559Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7690578Z 2025-12-04T09:48:03.7690676Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7690870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7690931Z res = mod(**inputs) 2025-12-04T09:48:03.7691177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7691264Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7691568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7691640Z outputs = layer_module( 2025-12-04T09:48:03.7691895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7691959Z outputs = self.rel_attn( 2025-12-04T09:48:03.7692230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7692334Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7692639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7692767Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7692773Z 2025-12-04T09:48:03.7692884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7693094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7693170Z res = mod(**inputs) 2025-12-04T09:48:03.7693445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7693536Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7693790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7693858Z outputs = layer_module( 2025-12-04T09:48:03.7694128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7694209Z outputs = self.rel_attn( 2025-12-04T09:48:03.7694453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7694550Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7694809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7694920Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7694923Z 2025-12-04T09:48:03.7695019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7695203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7695269Z res = mod(**inputs) 2025-12-04T09:48:03.7695526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7695613Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7695856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7695920Z outputs = layer_module( 2025-12-04T09:48:03.7696164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7696358Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7696609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7696682Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7696945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7697019Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7697255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7697321Z output = self.layer_1(output) 2025-12-04T09:48:03.7697333Z 2025-12-04T09:48:03.7697429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7697613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7697678Z res = mod(**inputs) 2025-12-04T09:48:03.7697920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7697995Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7698242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7698305Z outputs = layer_module( 2025-12-04T09:48:03.7698551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7698744Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7698989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7699068Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7699307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7699372Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7699614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7699698Z output = self.activation_function(output) 2025-12-04T09:48:03.7699927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7699995Z return self.act(input) 2025-12-04T09:48:03.7699999Z 2025-12-04T09:48:03.7700096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7700307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7700365Z res = mod(**inputs) 2025-12-04T09:48:03.7700615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7700691Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7700934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7701006Z outputs = layer_module( 2025-12-04T09:48:03.7701264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7701460Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7701717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7701792Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7702045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7702112Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7702353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7702429Z output = self.layer_2(output) 2025-12-04T09:48:03.7702450Z 2025-12-04T09:48:03.7702549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7702745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7702807Z res = mod(**inputs) 2025-12-04T09:48:03.7703044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7703128Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7703368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7703431Z outputs = layer_module( 2025-12-04T09:48:03.7703676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7703740Z outputs = self.rel_attn( 2025-12-04T09:48:03.7703985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7704079Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7704082Z 2025-12-04T09:48:03.7704181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7704373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7704435Z res = mod(**inputs) 2025-12-04T09:48:03.7704684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7704760Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7705000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7705070Z outputs = layer_module( 2025-12-04T09:48:03.7705307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7705373Z outputs = self.rel_attn( 2025-12-04T09:48:03.7705621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7705730Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7705734Z 2025-12-04T09:48:03.7705837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7706039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7706099Z res = mod(**inputs) 2025-12-04T09:48:03.7706344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7706419Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7706657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7706730Z outputs = layer_module( 2025-12-04T09:48:03.7706981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7707058Z outputs = self.rel_attn( 2025-12-04T09:48:03.7707296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7707367Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7707636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7707760Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7707763Z 2025-12-04T09:48:03.7707865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7708050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7708136Z res = mod(**inputs) 2025-12-04T09:48:03.7708385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7708461Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7708701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7708772Z outputs = layer_module( 2025-12-04T09:48:03.7709010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7709079Z outputs = self.rel_attn( 2025-12-04T09:48:03.7709313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7709435Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7709438Z 2025-12-04T09:48:03.7709540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7709726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7709790Z res = mod(**inputs) 2025-12-04T09:48:03.7710033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7710105Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7710352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7710416Z outputs = layer_module( 2025-12-04T09:48:03.7710656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7710724Z outputs = self.rel_attn( 2025-12-04T09:48:03.7710968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7711042Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7711312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7711445Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7711448Z 2025-12-04T09:48:03.7711551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7711750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7711815Z res = mod(**inputs) 2025-12-04T09:48:03.7712059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7712134Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7712381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7712446Z outputs = layer_module( 2025-12-04T09:48:03.7712699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7712773Z outputs = self.rel_attn( 2025-12-04T09:48:03.7713010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7713111Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7713114Z 2025-12-04T09:48:03.7713209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7713393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7713459Z res = mod(**inputs) 2025-12-04T09:48:03.7713699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7713784Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7714040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7714105Z outputs = layer_module( 2025-12-04T09:48:03.7714353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7714416Z outputs = self.rel_attn( 2025-12-04T09:48:03.7714657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7714732Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7714988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7715112Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7715116Z 2025-12-04T09:48:03.7715213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7715399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7715465Z res = mod(**inputs) 2025-12-04T09:48:03.7715707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7715783Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7716029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7716091Z outputs = layer_module( 2025-12-04T09:48:03.7716337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7716400Z outputs = self.rel_attn( 2025-12-04T09:48:03.7716638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7716732Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7716995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7717124Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7717128Z 2025-12-04T09:48:03.7717227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7717428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7717495Z res = mod(**inputs) 2025-12-04T09:48:03.7717733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7717810Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7718058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7718121Z outputs = layer_module( 2025-12-04T09:48:03.7718364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7718444Z outputs = self.rel_attn( 2025-12-04T09:48:03.7718688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7718780Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7719037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7719146Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7719150Z 2025-12-04T09:48:03.7719246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7719431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7719498Z res = mod(**inputs) 2025-12-04T09:48:03.7719757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7719834Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7720080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7720144Z outputs = layer_module( 2025-12-04T09:48:03.7720391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7720585Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7720829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7720910Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7721145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7721221Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7721463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7721530Z output = self.layer_1(output) 2025-12-04T09:48:03.7721535Z 2025-12-04T09:48:03.7721639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7721824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7721882Z res = mod(**inputs) 2025-12-04T09:48:03.7722131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7722208Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7722455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7722520Z outputs = layer_module( 2025-12-04T09:48:03.7722758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7722971Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7723232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7723314Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7723563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7723631Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7723879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7723964Z output = self.activation_function(output) 2025-12-04T09:48:03.7724188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7724263Z return self.act(input) 2025-12-04T09:48:03.7724268Z 2025-12-04T09:48:03.7724368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7724565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7724629Z res = mod(**inputs) 2025-12-04T09:48:03.7724875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7724959Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7725203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7725273Z outputs = layer_module( 2025-12-04T09:48:03.7725535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7725740Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7726000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7726073Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7726322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7726395Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7726641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7726718Z output = self.layer_2(output) 2025-12-04T09:48:03.7726721Z 2025-12-04T09:48:03.7726823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7727015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7727087Z res = mod(**inputs) 2025-12-04T09:48:03.7727341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7727428Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7727683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7727748Z outputs = layer_module( 2025-12-04T09:48:03.7728005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7728072Z outputs = self.rel_attn( 2025-12-04T09:48:03.7728321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7728428Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7728431Z 2025-12-04T09:48:03.7728535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7728752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7728817Z res = mod(**inputs) 2025-12-04T09:48:03.7729068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7729173Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7729425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7729491Z outputs = layer_module( 2025-12-04T09:48:03.7729760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7729827Z outputs = self.rel_attn( 2025-12-04T09:48:03.7730090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7730213Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7730219Z 2025-12-04T09:48:03.7730324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7730522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7730585Z res = mod(**inputs) 2025-12-04T09:48:03.7730846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7730926Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7731178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7731250Z outputs = layer_module( 2025-12-04T09:48:03.7731612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7731693Z outputs = self.rel_attn( 2025-12-04T09:48:03.7731982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7732059Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7732363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7732504Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7732508Z 2025-12-04T09:48:03.7732627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7732846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7732925Z res = mod(**inputs) 2025-12-04T09:48:03.7733183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7733263Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7733510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7733586Z outputs = layer_module( 2025-12-04T09:48:03.7733835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7733899Z outputs = self.rel_attn( 2025-12-04T09:48:03.7734152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7734280Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7734284Z 2025-12-04T09:48:03.7734392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7734581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7734644Z res = mod(**inputs) 2025-12-04T09:48:03.7734917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7734997Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7735248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7735332Z outputs = layer_module( 2025-12-04T09:48:03.7735579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7735649Z outputs = self.rel_attn( 2025-12-04T09:48:03.7735894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7735962Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7736235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7736374Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7736378Z 2025-12-04T09:48:03.7736485Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7736677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7736741Z res = mod(**inputs) 2025-12-04T09:48:03.7736998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7737075Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7737334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7737398Z outputs = layer_module( 2025-12-04T09:48:03.7737688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7737758Z outputs = self.rel_attn( 2025-12-04T09:48:03.7738004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7738097Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7738102Z 2025-12-04T09:48:03.7738207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7738397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7738465Z res = mod(**inputs) 2025-12-04T09:48:03.7738711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7738789Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7739041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7739107Z outputs = layer_module( 2025-12-04T09:48:03.7739351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7739422Z outputs = self.rel_attn( 2025-12-04T09:48:03.7739663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7739739Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7740002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7740123Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7740126Z 2025-12-04T09:48:03.7740231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7740420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7740490Z res = mod(**inputs) 2025-12-04T09:48:03.7740759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7740837Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7741091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7741172Z outputs = layer_module( 2025-12-04T09:48:03.7741427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7741499Z outputs = self.rel_attn( 2025-12-04T09:48:03.7741817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7741906Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7742173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7742288Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7742294Z 2025-12-04T09:48:03.7742402Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7742592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7742665Z res = mod(**inputs) 2025-12-04T09:48:03.7742912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7742991Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7743241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7743307Z outputs = layer_module( 2025-12-04T09:48:03.7743550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7743637Z outputs = self.rel_attn( 2025-12-04T09:48:03.7743879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7743970Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7744230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7744334Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7744338Z 2025-12-04T09:48:03.7744443Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7744635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7744703Z res = mod(**inputs) 2025-12-04T09:48:03.7744950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7745031Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7745288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7745352Z outputs = layer_module( 2025-12-04T09:48:03.7745597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7745804Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7746059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7746140Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7746392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7746460Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7746714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7746796Z output = self.layer_1(output) 2025-12-04T09:48:03.7746800Z 2025-12-04T09:48:03.7746904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7747105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7747163Z res = mod(**inputs) 2025-12-04T09:48:03.7747410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7747485Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7747725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7747799Z outputs = layer_module( 2025-12-04T09:48:03.7748039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7748254Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7748504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7748577Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7748828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7748894Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7749139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7749222Z output = self.activation_function(output) 2025-12-04T09:48:03.7749444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7749517Z return self.act(input) 2025-12-04T09:48:03.7749521Z 2025-12-04T09:48:03.7749619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7749803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7749870Z res = mod(**inputs) 2025-12-04T09:48:03.7750110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7750193Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7750429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7750492Z outputs = layer_module( 2025-12-04T09:48:03.7750738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7750931Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7751184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7751255Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7751497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7751570Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7751807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7751873Z output = self.layer_2(output) 2025-12-04T09:48:03.7751884Z 2025-12-04T09:48:03.7751978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7752163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7752230Z res = mod(**inputs) 2025-12-04T09:48:03.7752487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7752569Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7752820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7752905Z outputs = layer_module( 2025-12-04T09:48:03.7753152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7753217Z outputs = self.rel_attn( 2025-12-04T09:48:03.7753457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7753557Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7753562Z 2025-12-04T09:48:03.7753658Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7753859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7753930Z res = mod(**inputs) 2025-12-04T09:48:03.7754174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7754259Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7754497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7754559Z outputs = layer_module( 2025-12-04T09:48:03.7754804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7754868Z outputs = self.rel_attn( 2025-12-04T09:48:03.7755102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7755221Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7755226Z 2025-12-04T09:48:03.7755322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7755515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7755576Z res = mod(**inputs) 2025-12-04T09:48:03.7755815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7755896Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7756131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7756199Z outputs = layer_module( 2025-12-04T09:48:03.7756437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7756501Z outputs = self.rel_attn( 2025-12-04T09:48:03.7756746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7756814Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7757064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7757193Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7757196Z 2025-12-04T09:48:03.7757292Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7757486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7757545Z res = mod(**inputs) 2025-12-04T09:48:03.7757786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7757870Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7758115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7758203Z outputs = layer_module( 2025-12-04T09:48:03.7758444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7758526Z outputs = self.rel_attn( 2025-12-04T09:48:03.7758776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7758901Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7758905Z 2025-12-04T09:48:03.7759001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7759194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7759255Z res = mod(**inputs) 2025-12-04T09:48:03.7759526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7759606Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7759849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7759924Z outputs = layer_module( 2025-12-04T09:48:03.7760164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7760234Z outputs = self.rel_attn( 2025-12-04T09:48:03.7760474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7760539Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7760808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7760943Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7760948Z 2025-12-04T09:48:03.7761044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7761239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7761299Z res = mod(**inputs) 2025-12-04T09:48:03.7761553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7761630Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7761873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7761942Z outputs = layer_module( 2025-12-04T09:48:03.7762182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7762254Z outputs = self.rel_attn( 2025-12-04T09:48:03.7762501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7762593Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7762597Z 2025-12-04T09:48:03.7762699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7762886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7762946Z res = mod(**inputs) 2025-12-04T09:48:03.7763197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7763272Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7763524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7763588Z outputs = layer_module( 2025-12-04T09:48:03.7763833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7763917Z outputs = self.rel_attn( 2025-12-04T09:48:03.7764162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7764245Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7764514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7764632Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7764635Z 2025-12-04T09:48:03.7764739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7764925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7764988Z res = mod(**inputs) 2025-12-04T09:48:03.7765240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7765333Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7765580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7765643Z outputs = layer_module( 2025-12-04T09:48:03.7765879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7765951Z outputs = self.rel_attn( 2025-12-04T09:48:03.7766188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7766270Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7766532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7766653Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7766657Z 2025-12-04T09:48:03.7766764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7766949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7767009Z res = mod(**inputs) 2025-12-04T09:48:03.7767258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7767333Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7767579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7767640Z outputs = layer_module( 2025-12-04T09:48:03.7767878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7767949Z outputs = self.rel_attn( 2025-12-04T09:48:03.7768189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7768272Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7768536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7768640Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7768643Z 2025-12-04T09:48:03.7768746Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7768931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7768989Z res = mod(**inputs) 2025-12-04T09:48:03.7769240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7769316Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7769563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7769638Z outputs = layer_module( 2025-12-04T09:48:03.7769875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7770088Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7770335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7770408Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7770652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7770720Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7770963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7771044Z output = self.layer_1(output) 2025-12-04T09:48:03.7771048Z 2025-12-04T09:48:03.7771146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7771363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7771489Z res = mod(**inputs) 2025-12-04T09:48:03.7771778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7771863Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7772131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7772210Z outputs = layer_module( 2025-12-04T09:48:03.7772696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7772946Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7773216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7773300Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7773578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7773654Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7773921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7774022Z output = self.activation_function(output) 2025-12-04T09:48:03.7774249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7774331Z return self.act(input) 2025-12-04T09:48:03.7774335Z 2025-12-04T09:48:03.7774445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7774653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7774727Z res = mod(**inputs) 2025-12-04T09:48:03.7774996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7775080Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7775353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7775425Z outputs = layer_module( 2025-12-04T09:48:03.7775698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7775915Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7776224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7776316Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7776588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7776690Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7776963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7777037Z output = self.layer_2(output) 2025-12-04T09:48:03.7777040Z 2025-12-04T09:48:03.7777155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7777364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7777430Z res = mod(**inputs) 2025-12-04T09:48:03.7777735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7777825Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7778105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7778177Z outputs = layer_module( 2025-12-04T09:48:03.7778443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7778522Z outputs = self.rel_attn( 2025-12-04T09:48:03.7778789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7778902Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7778906Z 2025-12-04T09:48:03.7779014Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7779239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7779313Z res = mod(**inputs) 2025-12-04T09:48:03.7779586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7779670Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7779955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7780020Z outputs = layer_module( 2025-12-04T09:48:03.7780277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7780342Z outputs = self.rel_attn( 2025-12-04T09:48:03.7780591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7780697Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7780700Z 2025-12-04T09:48:03.7780799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7780995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7781054Z res = mod(**inputs) 2025-12-04T09:48:03.7781304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7781389Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7781635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7781700Z outputs = layer_module( 2025-12-04T09:48:03.7781950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7782019Z outputs = self.rel_attn( 2025-12-04T09:48:03.7782267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7782336Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7782615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7782764Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7782768Z 2025-12-04T09:48:03.7782867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7783064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7783124Z res = mod(**inputs) 2025-12-04T09:48:03.7783368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7783452Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7783708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7783784Z outputs = layer_module( 2025-12-04T09:48:03.7784029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7784093Z outputs = self.rel_attn( 2025-12-04T09:48:03.7784336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7784457Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7784460Z 2025-12-04T09:48:03.7784555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7784744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7784802Z res = mod(**inputs) 2025-12-04T09:48:03.7785064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7785152Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7785397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7785467Z outputs = layer_module( 2025-12-04T09:48:03.7785709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7785774Z outputs = self.rel_attn( 2025-12-04T09:48:03.7786022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7786090Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7786356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7786481Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7786485Z 2025-12-04T09:48:03.7786584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7786783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7786842Z res = mod(**inputs) 2025-12-04T09:48:03.7787080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7787161Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7787396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7787465Z outputs = layer_module( 2025-12-04T09:48:03.7787704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7787768Z outputs = self.rel_attn( 2025-12-04T09:48:03.7788017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7788129Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7788133Z 2025-12-04T09:48:03.7788237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7788439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7788499Z res = mod(**inputs) 2025-12-04T09:48:03.7788746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7788822Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7789063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7789132Z outputs = layer_module( 2025-12-04T09:48:03.7789374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7789459Z outputs = self.rel_attn( 2025-12-04T09:48:03.7789698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7789765Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7790027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7790141Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7790144Z 2025-12-04T09:48:03.7790245Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7790428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7790487Z res = mod(**inputs) 2025-12-04T09:48:03.7790736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7790827Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7791066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7791136Z outputs = layer_module( 2025-12-04T09:48:03.7791375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7791456Z outputs = self.rel_attn( 2025-12-04T09:48:03.7791691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7791774Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7792039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7792148Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7792152Z 2025-12-04T09:48:03.7792253Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7792438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7792497Z res = mod(**inputs) 2025-12-04T09:48:03.7792743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7792821Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7793058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7793126Z outputs = layer_module( 2025-12-04T09:48:03.7793363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7793432Z outputs = self.rel_attn( 2025-12-04T09:48:03.7793668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7793752Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7794029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7794148Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7794151Z 2025-12-04T09:48:03.7794247Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7794438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7794499Z res = mod(**inputs) 2025-12-04T09:48:03.7794745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7794819Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7795059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7795145Z outputs = layer_module( 2025-12-04T09:48:03.7795384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7795586Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7795838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7795910Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7796157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7796224Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7796461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7796562Z output = self.layer_1(output) 2025-12-04T09:48:03.7796565Z 2025-12-04T09:48:03.7796664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7796858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7796921Z res = mod(**inputs) 2025-12-04T09:48:03.7797161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7797245Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7797490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7797562Z outputs = layer_module( 2025-12-04T09:48:03.7797801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7797995Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7798252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7798324Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7798575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7798649Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7798896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7798986Z output = self.activation_function(output) 2025-12-04T09:48:03.7799193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7799262Z return self.act(input) 2025-12-04T09:48:03.7799265Z 2025-12-04T09:48:03.7799374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7799583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7799653Z res = mod(**inputs) 2025-12-04T09:48:03.7799900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7799993Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7800261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7800326Z outputs = layer_module( 2025-12-04T09:48:03.7800566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7800764Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7801037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7801120Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7801369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7801438Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7801693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7801762Z output = self.layer_2(output) 2025-12-04T09:48:03.7801765Z 2025-12-04T09:48:03.7801871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7802063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7802123Z res = mod(**inputs) 2025-12-04T09:48:03.7802394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7802473Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7802725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7802799Z outputs = layer_module( 2025-12-04T09:48:03.7803046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7803117Z outputs = self.rel_attn( 2025-12-04T09:48:03.7803363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7803459Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7803462Z 2025-12-04T09:48:03.7803568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7803758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7803827Z res = mod(**inputs) 2025-12-04T09:48:03.7804078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7804158Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7804411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7804478Z outputs = layer_module( 2025-12-04T09:48:03.7804723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7804799Z outputs = self.rel_attn( 2025-12-04T09:48:03.7805041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7805142Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7805147Z 2025-12-04T09:48:03.7805246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7805451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7805523Z res = mod(**inputs) 2025-12-04T09:48:03.7805769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7805861Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7806117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7806180Z outputs = layer_module( 2025-12-04T09:48:03.7806433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7806496Z outputs = self.rel_attn( 2025-12-04T09:48:03.7806742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7806820Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7807098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7807230Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7807235Z 2025-12-04T09:48:03.7807335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7807525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7807593Z res = mod(**inputs) 2025-12-04T09:48:03.7807842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7807919Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7808172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7808252Z outputs = layer_module( 2025-12-04T09:48:03.7808506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7808574Z outputs = self.rel_attn( 2025-12-04T09:48:03.7808816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7808953Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7808957Z 2025-12-04T09:48:03.7809057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7809255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7809317Z res = mod(**inputs) 2025-12-04T09:48:03.7809562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7809652Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7809902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7809967Z outputs = layer_module( 2025-12-04T09:48:03.7810220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7810288Z outputs = self.rel_attn( 2025-12-04T09:48:03.7810538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7810606Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7810870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7810998Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7811003Z 2025-12-04T09:48:03.7811101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7811314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7811378Z res = mod(**inputs) 2025-12-04T09:48:03.7811696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7811811Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7812056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7812119Z outputs = layer_module( 2025-12-04T09:48:03.7812378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7812445Z outputs = self.rel_attn( 2025-12-04T09:48:03.7812699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7812814Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7812820Z 2025-12-04T09:48:03.7812931Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7813146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7813214Z res = mod(**inputs) 2025-12-04T09:48:03.7813494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7813578Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7813847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7813926Z outputs = layer_module( 2025-12-04T09:48:03.7814196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7814283Z outputs = self.rel_attn( 2025-12-04T09:48:03.7814552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7814622Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7814905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7815028Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7815032Z 2025-12-04T09:48:03.7815133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7815339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7815400Z res = mod(**inputs) 2025-12-04T09:48:03.7815666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7815747Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7816009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7816082Z outputs = layer_module( 2025-12-04T09:48:03.7816343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7816410Z outputs = self.rel_attn( 2025-12-04T09:48:03.7816675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7816761Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7817051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7817163Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7817168Z 2025-12-04T09:48:03.7817270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7817478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7817560Z res = mod(**inputs) 2025-12-04T09:48:03.7817816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7817920Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7818174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7818249Z outputs = layer_module( 2025-12-04T09:48:03.7818501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7818568Z outputs = self.rel_attn( 2025-12-04T09:48:03.7818830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7818921Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7819225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7819337Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7819342Z 2025-12-04T09:48:03.7819443Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7819646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7819708Z res = mod(**inputs) 2025-12-04T09:48:03.7819964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7820053Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7820310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7820403Z outputs = layer_module( 2025-12-04T09:48:03.7820665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7820871Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7821147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7821222Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7821491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7821560Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7821821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7821901Z output = self.layer_1(output) 2025-12-04T09:48:03.7821904Z 2025-12-04T09:48:03.7822006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7822207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7822277Z res = mod(**inputs) 2025-12-04T09:48:03.7822589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7822683Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7822945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7823011Z outputs = layer_module( 2025-12-04T09:48:03.7823280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7823485Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7823762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7823854Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7824113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7824209Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7824473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7824559Z output = self.activation_function(output) 2025-12-04T09:48:03.7824784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7824853Z return self.act(input) 2025-12-04T09:48:03.7824856Z 2025-12-04T09:48:03.7824968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7825169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7825248Z res = mod(**inputs) 2025-12-04T09:48:03.7825518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7825602Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7825861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7825929Z outputs = layer_module( 2025-12-04T09:48:03.7826179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7826387Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7826645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7826736Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7827014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7827083Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7827355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7827424Z output = self.layer_2(output) 2025-12-04T09:48:03.7827427Z 2025-12-04T09:48:03.7827527Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7827730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7827792Z res = mod(**inputs) 2025-12-04T09:48:03.7828055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7828136Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7828401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7828475Z outputs = layer_module( 2025-12-04T09:48:03.7828729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7828797Z outputs = self.rel_attn( 2025-12-04T09:48:03.7829059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7829156Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7829160Z 2025-12-04T09:48:03.7829266Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7829461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7829524Z res = mod(**inputs) 2025-12-04T09:48:03.7829792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7829889Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7830149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7830238Z outputs = layer_module( 2025-12-04T09:48:03.7830500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7830572Z outputs = self.rel_attn( 2025-12-04T09:48:03.7830818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7830913Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7830917Z 2025-12-04T09:48:03.7831025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7831217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7831327Z res = mod(**inputs) 2025-12-04T09:48:03.7831577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7831657Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7831909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7831973Z outputs = layer_module( 2025-12-04T09:48:03.7832217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7832288Z outputs = self.rel_attn( 2025-12-04T09:48:03.7832528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7832620Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7832894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7833019Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7833022Z 2025-12-04T09:48:03.7833133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7833327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7833396Z res = mod(**inputs) 2025-12-04T09:48:03.7833650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7833728Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7833994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7834061Z outputs = layer_module( 2025-12-04T09:48:03.7834317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7834391Z outputs = self.rel_attn( 2025-12-04T09:48:03.7834643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7834778Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7834781Z 2025-12-04T09:48:03.7834880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7835072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7835141Z res = mod(**inputs) 2025-12-04T09:48:03.7835398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7835487Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7835744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7835828Z outputs = layer_module( 2025-12-04T09:48:03.7836082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7836162Z outputs = self.rel_attn( 2025-12-04T09:48:03.7836406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7836481Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7836740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7836870Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7836873Z 2025-12-04T09:48:03.7836974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7837159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7837242Z res = mod(**inputs) 2025-12-04T09:48:03.7837494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7869317Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7869830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7869913Z outputs = layer_module( 2025-12-04T09:48:03.7870185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7870269Z outputs = self.rel_attn( 2025-12-04T09:48:03.7870523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7870758Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7870768Z 2025-12-04T09:48:03.7870898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7871107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7871184Z res = mod(**inputs) 2025-12-04T09:48:03.7871454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7871543Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7871807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7871876Z outputs = layer_module( 2025-12-04T09:48:03.7872138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7872215Z outputs = self.rel_attn( 2025-12-04T09:48:03.7872694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7872790Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7873087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7873228Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7873233Z 2025-12-04T09:48:03.7873360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7873574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7873651Z res = mod(**inputs) 2025-12-04T09:48:03.7873931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7874017Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7874293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7874365Z outputs = layer_module( 2025-12-04T09:48:03.7874686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7874823Z outputs = self.rel_attn( 2025-12-04T09:48:03.7875087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7875190Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7875479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7875597Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7875601Z 2025-12-04T09:48:03.7875724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7875928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7876048Z res = mod(**inputs) 2025-12-04T09:48:03.7876308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7876392Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7876657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7876726Z outputs = layer_module( 2025-12-04T09:48:03.7876978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7877055Z outputs = self.rel_attn( 2025-12-04T09:48:03.7877306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7877426Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7877703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7877817Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7877822Z 2025-12-04T09:48:03.7877936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7878135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7878207Z res = mod(**inputs) 2025-12-04T09:48:03.7878475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7878556Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7878818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7878887Z outputs = layer_module( 2025-12-04T09:48:03.7879163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7879377Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7879633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7879719Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7879965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7880034Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7880291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7880363Z output = self.layer_1(output) 2025-12-04T09:48:03.7880369Z 2025-12-04T09:48:03.7880475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7880669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7880748Z res = mod(**inputs) 2025-12-04T09:48:03.7881010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7881109Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7881365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7881440Z outputs = layer_module( 2025-12-04T09:48:03.7881697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7881913Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7882181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7882269Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7882530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7882602Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7882858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7882945Z output = self.activation_function(output) 2025-12-04T09:48:03.7883161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7883240Z return self.act(input) 2025-12-04T09:48:03.7883243Z 2025-12-04T09:48:03.7883347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7883561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7883633Z res = mod(**inputs) 2025-12-04T09:48:03.7883896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7883985Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7884234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7884299Z outputs = layer_module( 2025-12-04T09:48:03.7884553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7884749Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7885016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7885095Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7885353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7885433Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7885683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7885756Z output = self.layer_2(output) 2025-12-04T09:48:03.7885768Z 2025-12-04T09:48:03.7885871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7886066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7886136Z res = mod(**inputs) 2025-12-04T09:48:03.7886394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7886476Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7886740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7886822Z outputs = layer_module( 2025-12-04T09:48:03.7887083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7887168Z outputs = self.rel_attn( 2025-12-04T09:48:03.7887420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7887526Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7887529Z 2025-12-04T09:48:03.7887628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7887822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7887890Z res = mod(**inputs) 2025-12-04T09:48:03.7888147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7888251Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7888507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7888575Z outputs = layer_module( 2025-12-04T09:48:03.7888836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7888905Z outputs = self.rel_attn( 2025-12-04T09:48:03.7889154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7889264Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7889268Z 2025-12-04T09:48:03.7889368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7889587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7889651Z res = mod(**inputs) 2025-12-04T09:48:03.7889907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7889996Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7890249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7890321Z outputs = layer_module( 2025-12-04T09:48:03.7890569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7890635Z outputs = self.rel_attn( 2025-12-04T09:48:03.7890892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7890964Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7891242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7891396Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7891400Z 2025-12-04T09:48:03.7891593Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7891819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7891886Z res = mod(**inputs) 2025-12-04T09:48:03.7892164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7892258Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7892543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7892618Z outputs = layer_module( 2025-12-04T09:48:03.7892868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7892937Z outputs = self.rel_attn( 2025-12-04T09:48:03.7893213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7893351Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7893372Z 2025-12-04T09:48:03.7893482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7893675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7893739Z res = mod(**inputs) 2025-12-04T09:48:03.7894000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7894082Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7894336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7894414Z outputs = layer_module( 2025-12-04T09:48:03.7894683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7894759Z outputs = self.rel_attn( 2025-12-04T09:48:03.7895014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7895085Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7895361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7895488Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7895492Z 2025-12-04T09:48:03.7895594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7895812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7895875Z res = mod(**inputs) 2025-12-04T09:48:03.7896141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7896220Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7896474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7896545Z outputs = layer_module( 2025-12-04T09:48:03.7896812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7896883Z outputs = self.rel_attn( 2025-12-04T09:48:03.7897123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7897224Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7897229Z 2025-12-04T09:48:03.7897325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7897514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7897582Z res = mod(**inputs) 2025-12-04T09:48:03.7897822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7897908Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7898147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7898209Z outputs = layer_module( 2025-12-04T09:48:03.7898455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7898518Z outputs = self.rel_attn( 2025-12-04T09:48:03.7898761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7898839Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7899117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7899244Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7899263Z 2025-12-04T09:48:03.7899362Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7899544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7899612Z res = mod(**inputs) 2025-12-04T09:48:03.7899852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7899927Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7900171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7900237Z outputs = layer_module( 2025-12-04T09:48:03.7900499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7900566Z outputs = self.rel_attn( 2025-12-04T09:48:03.7900803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7900901Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7901161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7901274Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7901278Z 2025-12-04T09:48:03.7901371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7901554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7901637Z res = mod(**inputs) 2025-12-04T09:48:03.7901881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7901957Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7902205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7902271Z outputs = layer_module( 2025-12-04T09:48:03.7902516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7902577Z outputs = self.rel_attn( 2025-12-04T09:48:03.7902816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7902905Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7903160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7903275Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7903279Z 2025-12-04T09:48:03.7903374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7903556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7903623Z res = mod(**inputs) 2025-12-04T09:48:03.7903865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7903939Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7904187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7904249Z outputs = layer_module( 2025-12-04T09:48:03.7904497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7904708Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7904957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7905058Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7905302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7905379Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7905620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7905689Z output = self.layer_1(output) 2025-12-04T09:48:03.7905692Z 2025-12-04T09:48:03.7905799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7905988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7906064Z res = mod(**inputs) 2025-12-04T09:48:03.7906319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7906396Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7906645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7906708Z outputs = layer_module( 2025-12-04T09:48:03.7906950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7907149Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7907395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7907496Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7907745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7907812Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7908064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7908146Z output = self.activation_function(output) 2025-12-04T09:48:03.7908352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7908422Z return self.act(input) 2025-12-04T09:48:03.7908426Z 2025-12-04T09:48:03.7908523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7908720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7908781Z res = mod(**inputs) 2025-12-04T09:48:03.7909028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7909113Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7909358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7909430Z outputs = layer_module( 2025-12-04T09:48:03.7909673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7909868Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7910122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7910195Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7910439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7910514Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7910773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7910863Z output = self.layer_2(output) 2025-12-04T09:48:03.7910866Z 2025-12-04T09:48:03.7910963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7911149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7911217Z res = mod(**inputs) 2025-12-04T09:48:03.7911458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7911541Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7911780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7911849Z outputs = layer_module( 2025-12-04T09:48:03.7912398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7912469Z outputs = self.rel_attn( 2025-12-04T09:48:03.7912716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7912811Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7912814Z 2025-12-04T09:48:03.7912917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7913102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7913162Z res = mod(**inputs) 2025-12-04T09:48:03.7913411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7913507Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7913760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7913825Z outputs = layer_module( 2025-12-04T09:48:03.7914061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7914139Z outputs = self.rel_attn( 2025-12-04T09:48:03.7914376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7914472Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7914475Z 2025-12-04T09:48:03.7914583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7914768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7914836Z res = mod(**inputs) 2025-12-04T09:48:03.7915076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7915154Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7915398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7915462Z outputs = layer_module( 2025-12-04T09:48:03.7915701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7915771Z outputs = self.rel_attn( 2025-12-04T09:48:03.7916008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7916083Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7916344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7916470Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7916474Z 2025-12-04T09:48:03.7916594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7916784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7916864Z res = mod(**inputs) 2025-12-04T09:48:03.7917121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7917195Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7917447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7917510Z outputs = layer_module( 2025-12-04T09:48:03.7917749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7917820Z outputs = self.rel_attn( 2025-12-04T09:48:03.7918075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7918210Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7918215Z 2025-12-04T09:48:03.7918312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7918503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7918567Z res = mod(**inputs) 2025-12-04T09:48:03.7918799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7918879Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7919110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7919205Z outputs = layer_module( 2025-12-04T09:48:03.7919451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7919512Z outputs = self.rel_attn( 2025-12-04T09:48:03.7919748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7919823Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7920079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7920204Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7920207Z 2025-12-04T09:48:03.7920301Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7920480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7920547Z res = mod(**inputs) 2025-12-04T09:48:03.7920788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7920870Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7921108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7921169Z outputs = layer_module( 2025-12-04T09:48:03.7921413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7921472Z outputs = self.rel_attn( 2025-12-04T09:48:03.7921707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7921807Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7921811Z 2025-12-04T09:48:03.7921904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7922097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7922154Z res = mod(**inputs) 2025-12-04T09:48:03.7922407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7922506Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7922739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7922807Z outputs = layer_module( 2025-12-04T09:48:03.7923039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7923100Z outputs = self.rel_attn( 2025-12-04T09:48:03.7923336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7923404Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7923672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7923796Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7923800Z 2025-12-04T09:48:03.7923895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7924081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7924138Z res = mod(**inputs) 2025-12-04T09:48:03.7924372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7924451Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7924682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7924761Z outputs = layer_module( 2025-12-04T09:48:03.7925001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7925066Z outputs = self.rel_attn( 2025-12-04T09:48:03.7925306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7925391Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7925648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7925762Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7925765Z 2025-12-04T09:48:03.7925862Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7926054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7926115Z res = mod(**inputs) 2025-12-04T09:48:03.7926357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7926440Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7926679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7926745Z outputs = layer_module( 2025-12-04T09:48:03.7926991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7927053Z outputs = self.rel_attn( 2025-12-04T09:48:03.7927297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7927379Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7927637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7927750Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7927754Z 2025-12-04T09:48:03.7927867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7928069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7928144Z res = mod(**inputs) 2025-12-04T09:48:03.7928377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7928459Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7928696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7928759Z outputs = layer_module( 2025-12-04T09:48:03.7929002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7929216Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7929473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7929547Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7929789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7929865Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7930102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7930175Z output = self.layer_1(output) 2025-12-04T09:48:03.7930178Z 2025-12-04T09:48:03.7930274Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7930460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7930544Z res = mod(**inputs) 2025-12-04T09:48:03.7930794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7930870Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7931119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7931183Z outputs = layer_module( 2025-12-04T09:48:03.7931503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7931707Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7931954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7932036Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7932322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7932405Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7932673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7932778Z output = self.activation_function(output) 2025-12-04T09:48:03.7933002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7933071Z return self.act(input) 2025-12-04T09:48:03.7933075Z 2025-12-04T09:48:03.7933176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7933382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7933444Z res = mod(**inputs) 2025-12-04T09:48:03.7933711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7933789Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7934050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7934139Z outputs = layer_module( 2025-12-04T09:48:03.7934382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7934584Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7934832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7934903Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7935152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7935222Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7935477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7935558Z output = self.layer_2(output) 2025-12-04T09:48:03.7935563Z 2025-12-04T09:48:03.7935659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7935851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7935911Z res = mod(**inputs) 2025-12-04T09:48:03.7936149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7936233Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7936473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7936561Z outputs = layer_module( 2025-12-04T09:48:03.7936806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7936873Z outputs = self.rel_attn( 2025-12-04T09:48:03.7937118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7937211Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7937215Z 2025-12-04T09:48:03.7937307Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7937498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7937557Z res = mod(**inputs) 2025-12-04T09:48:03.7937806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7937883Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7938123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7938195Z outputs = layer_module( 2025-12-04T09:48:03.7938434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7938499Z outputs = self.rel_attn( 2025-12-04T09:48:03.7938742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7938835Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7938838Z 2025-12-04T09:48:03.7938941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7939131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7939191Z res = mod(**inputs) 2025-12-04T09:48:03.7939446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7939533Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7939790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7939867Z outputs = layer_module( 2025-12-04T09:48:03.7940113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7940184Z outputs = self.rel_attn( 2025-12-04T09:48:03.7940429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7940496Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7940770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7940893Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7940896Z 2025-12-04T09:48:03.7941024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7941210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7941269Z res = mod(**inputs) 2025-12-04T09:48:03.7941517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7941591Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7941834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7941896Z outputs = layer_module( 2025-12-04T09:48:03.7942131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7942220Z outputs = self.rel_attn( 2025-12-04T09:48:03.7942465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7942591Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7942601Z 2025-12-04T09:48:03.7942697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7942884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7942949Z res = mod(**inputs) 2025-12-04T09:48:03.7943191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7943266Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7943514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7943578Z outputs = layer_module( 2025-12-04T09:48:03.7943823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7943885Z outputs = self.rel_attn( 2025-12-04T09:48:03.7944126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7944199Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7944465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7944592Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7944596Z 2025-12-04T09:48:03.7944690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7944875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7944940Z res = mod(**inputs) 2025-12-04T09:48:03.7945184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7945265Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7945518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7945596Z outputs = layer_module( 2025-12-04T09:48:03.7945842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7945906Z outputs = self.rel_attn( 2025-12-04T09:48:03.7946141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7946240Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7946244Z 2025-12-04T09:48:03.7946336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7946529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7946587Z res = mod(**inputs) 2025-12-04T09:48:03.7946844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7946932Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7947179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7947249Z outputs = layer_module( 2025-12-04T09:48:03.7947488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7947553Z outputs = self.rel_attn( 2025-12-04T09:48:03.7947800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7947883Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7948143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7948270Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7948274Z 2025-12-04T09:48:03.7948372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7948567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7948627Z res = mod(**inputs) 2025-12-04T09:48:03.7948871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7948956Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7949202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7949270Z outputs = layer_module( 2025-12-04T09:48:03.7949512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7949577Z outputs = self.rel_attn( 2025-12-04T09:48:03.7949828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7949912Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7950172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7950284Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7950288Z 2025-12-04T09:48:03.7950382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7950575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7950633Z res = mod(**inputs) 2025-12-04T09:48:03.7950879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7950963Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7951220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7951290Z outputs = layer_module( 2025-12-04T09:48:03.7951544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7951607Z outputs = self.rel_attn( 2025-12-04T09:48:03.7951847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7951927Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7952183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7952295Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7952298Z 2025-12-04T09:48:03.7952409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7952603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7952662Z res = mod(**inputs) 2025-12-04T09:48:03.7952907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7952991Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7953237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7953308Z outputs = layer_module( 2025-12-04T09:48:03.7953549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7953758Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7954015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7954088Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7954329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7954404Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7954641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7954714Z output = self.layer_1(output) 2025-12-04T09:48:03.7954718Z 2025-12-04T09:48:03.7954812Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7954995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7955063Z res = mod(**inputs) 2025-12-04T09:48:03.7955308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7955393Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7955634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7955699Z outputs = layer_module( 2025-12-04T09:48:03.7955946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7956137Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7956385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7956464Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7956706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7956780Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7957029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7957164Z output = self.activation_function(output) 2025-12-04T09:48:03.7957380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7957447Z return self.act(input) 2025-12-04T09:48:03.7957450Z 2025-12-04T09:48:03.7957555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7957744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7957804Z res = mod(**inputs) 2025-12-04T09:48:03.7958063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7958139Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7958395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7958469Z outputs = layer_module( 2025-12-04T09:48:03.7958713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7958915Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7959172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7959242Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7959502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7959589Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7959849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7959921Z output = self.layer_2(output) 2025-12-04T09:48:03.7959924Z 2025-12-04T09:48:03.7960025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7960227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7960288Z res = mod(**inputs) 2025-12-04T09:48:03.7960542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7960626Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7960881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7960953Z outputs = layer_module( 2025-12-04T09:48:03.7961200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7961268Z outputs = self.rel_attn( 2025-12-04T09:48:03.7961527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7961624Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7961628Z 2025-12-04T09:48:03.7961732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7961924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7961983Z res = mod(**inputs) 2025-12-04T09:48:03.7962243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7962319Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7962603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7962677Z outputs = layer_module( 2025-12-04T09:48:03.7962962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7963036Z outputs = self.rel_attn( 2025-12-04T09:48:03.7963311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7963407Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7963410Z 2025-12-04T09:48:03.7963514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7963702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7963761Z res = mod(**inputs) 2025-12-04T09:48:03.7964015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7964093Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7964365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7964431Z outputs = layer_module( 2025-12-04T09:48:03.7964675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7964747Z outputs = self.rel_attn( 2025-12-04T09:48:03.7964993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7965067Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7965330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7965454Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7965473Z 2025-12-04T09:48:03.7965580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7965773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7965831Z res = mod(**inputs) 2025-12-04T09:48:03.7966090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7966169Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7966428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7966490Z outputs = layer_module( 2025-12-04T09:48:03.7966739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7966812Z outputs = self.rel_attn( 2025-12-04T09:48:03.7967068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7967198Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7967203Z 2025-12-04T09:48:03.7967301Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7967496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7967564Z res = mod(**inputs) 2025-12-04T09:48:03.7967820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7967896Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7968157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7968220Z outputs = layer_module( 2025-12-04T09:48:03.7968478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7968543Z outputs = self.rel_attn( 2025-12-04T09:48:03.7968809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7968887Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7969175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7969305Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7969309Z 2025-12-04T09:48:03.7969406Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7969594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7969661Z res = mod(**inputs) 2025-12-04T09:48:03.7969908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7969986Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7970257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7970326Z outputs = layer_module( 2025-12-04T09:48:03.7970586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7970653Z outputs = self.rel_attn( 2025-12-04T09:48:03.7970906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7971012Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7971016Z 2025-12-04T09:48:03.7971114Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7971316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7971397Z res = mod(**inputs) 2025-12-04T09:48:03.7971733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7971829Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7972084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7972151Z outputs = layer_module( 2025-12-04T09:48:03.7972562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7972641Z outputs = self.rel_attn( 2025-12-04T09:48:03.7972943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7973020Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7973326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7973470Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7973474Z 2025-12-04T09:48:03.7973590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7973827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7973892Z res = mod(**inputs) 2025-12-04T09:48:03.7974149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7974237Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7974501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7974566Z outputs = layer_module( 2025-12-04T09:48:03.7974822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7974891Z outputs = self.rel_attn( 2025-12-04T09:48:03.7975187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7975275Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7975569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7975684Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7975687Z 2025-12-04T09:48:03.7975784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7975974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7976042Z res = mod(**inputs) 2025-12-04T09:48:03.7976286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7976372Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7976640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7976706Z outputs = layer_module( 2025-12-04T09:48:03.7976960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7977027Z outputs = self.rel_attn( 2025-12-04T09:48:03.7977276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7977360Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.7977622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.7977734Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.7977759Z 2025-12-04T09:48:03.7977857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7978049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7978118Z res = mod(**inputs) 2025-12-04T09:48:03.7978366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7978453Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7978699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7978763Z outputs = layer_module( 2025-12-04T09:48:03.7979017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7979217Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7979480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7979557Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7979805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7979883Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7980127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.7980195Z output = self.layer_1(output) 2025-12-04T09:48:03.7980206Z 2025-12-04T09:48:03.7980303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7980493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7980560Z res = mod(**inputs) 2025-12-04T09:48:03.7980807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7980886Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7981159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7981226Z outputs = layer_module( 2025-12-04T09:48:03.7981496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7981693Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7981943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7982022Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7982266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7982336Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7982603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.7982689Z output = self.activation_function(output) 2025-12-04T09:48:03.7982902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.7982968Z return self.act(input) 2025-12-04T09:48:03.7982972Z 2025-12-04T09:48:03.7983069Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7983262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7983323Z res = mod(**inputs) 2025-12-04T09:48:03.7983574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7983666Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7983912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7983985Z outputs = layer_module( 2025-12-04T09:48:03.7984231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.7984428Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.7984695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.7984765Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.7985013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.7985080Z output_x = self.ff(output_x) 2025-12-04T09:48:03.7985326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.7985403Z output = self.layer_2(output) 2025-12-04T09:48:03.7985408Z 2025-12-04T09:48:03.7985506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7985699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7985761Z res = mod(**inputs) 2025-12-04T09:48:03.7986010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7986094Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7986339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7986403Z outputs = layer_module( 2025-12-04T09:48:03.7986657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7986726Z outputs = self.rel_attn( 2025-12-04T09:48:03.7987008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.7987107Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.7987125Z 2025-12-04T09:48:03.7987227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7987437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7987499Z res = mod(**inputs) 2025-12-04T09:48:03.7987757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7987837Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7988083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7988159Z outputs = layer_module( 2025-12-04T09:48:03.7988417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7988483Z outputs = self.rel_attn( 2025-12-04T09:48:03.7988735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.7988830Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.7988833Z 2025-12-04T09:48:03.7988933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7989118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7989176Z res = mod(**inputs) 2025-12-04T09:48:03.7989427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7989518Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7989761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7989832Z outputs = layer_module( 2025-12-04T09:48:03.7990068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7990140Z outputs = self.rel_attn( 2025-12-04T09:48:03.7990380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7990446Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7990710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.7990833Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.7990838Z 2025-12-04T09:48:03.7990940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7991128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7991188Z res = mod(**inputs) 2025-12-04T09:48:03.7991438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7991513Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7991754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7991822Z outputs = layer_module( 2025-12-04T09:48:03.7992061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7992129Z outputs = self.rel_attn( 2025-12-04T09:48:03.7992370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.7992492Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.7992498Z 2025-12-04T09:48:03.7992619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7992809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7992891Z res = mod(**inputs) 2025-12-04T09:48:03.7993137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7993212Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7993467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7993531Z outputs = layer_module( 2025-12-04T09:48:03.7993775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7993848Z outputs = self.rel_attn( 2025-12-04T09:48:03.7994113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7994193Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7994455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.7994579Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.7994582Z 2025-12-04T09:48:03.7994686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7994875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7994953Z res = mod(**inputs) 2025-12-04T09:48:03.7995194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7995286Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7995533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7995596Z outputs = layer_module( 2025-12-04T09:48:03.7995832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7995903Z outputs = self.rel_attn( 2025-12-04T09:48:03.7996136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.7996235Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.7996238Z 2025-12-04T09:48:03.7996332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7996512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7996578Z res = mod(**inputs) 2025-12-04T09:48:03.7996815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7996898Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7997138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7997201Z outputs = layer_module( 2025-12-04T09:48:03.7997443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7997504Z outputs = self.rel_attn( 2025-12-04T09:48:03.7997738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.7997812Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.7998063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.7998189Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.7998192Z 2025-12-04T09:48:03.7998287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.7998482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.7998562Z res = mod(**inputs) 2025-12-04T09:48:03.7998804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.7998880Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.7999126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.7999190Z outputs = layer_module( 2025-12-04T09:48:03.7999440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.7999508Z outputs = self.rel_attn( 2025-12-04T09:48:03.7999770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.7999866Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8000129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8000247Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8000251Z 2025-12-04T09:48:03.8000351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8000539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8000607Z res = mod(**inputs) 2025-12-04T09:48:03.8000853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8000949Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8001201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8001268Z outputs = layer_module( 2025-12-04T09:48:03.8001518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8001594Z outputs = self.rel_attn( 2025-12-04T09:48:03.8001830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8001919Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8002177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8002289Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8002292Z 2025-12-04T09:48:03.8002393Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8002584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8002651Z res = mod(**inputs) 2025-12-04T09:48:03.8002896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8002975Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8003234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8003298Z outputs = layer_module( 2025-12-04T09:48:03.8003546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8003744Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8003999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8004082Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8004343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8004421Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8004700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.8004768Z output = self.layer_1(output) 2025-12-04T09:48:03.8004771Z 2025-12-04T09:48:03.8004873Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8005061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8005123Z res = mod(**inputs) 2025-12-04T09:48:03.8005383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8005463Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8005741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8005809Z outputs = layer_module( 2025-12-04T09:48:03.8006062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8006276Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8006535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8006616Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8006879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8006977Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8007241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.8007327Z output = self.activation_function(output) 2025-12-04T09:48:03.8007541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.8007618Z return self.act(input) 2025-12-04T09:48:03.8007622Z 2025-12-04T09:48:03.8007720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8007921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8007982Z res = mod(**inputs) 2025-12-04T09:48:03.8008238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8008324Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8008580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8008654Z outputs = layer_module( 2025-12-04T09:48:03.8008908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8009109Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8009378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8009450Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8009705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8009779Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8010033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.8010109Z output = self.layer_2(output) 2025-12-04T09:48:03.8010114Z 2025-12-04T09:48:03.8010232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8010430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8010516Z res = mod(**inputs) 2025-12-04T09:48:03.8010772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8010857Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8011114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8011178Z outputs = layer_module( 2025-12-04T09:48:03.8011544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8011631Z outputs = self.rel_attn( 2025-12-04T09:48:03.8011917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.8012034Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.8012039Z 2025-12-04T09:48:03.8012147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8012362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8012428Z res = mod(**inputs) 2025-12-04T09:48:03.8012696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8012788Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8013058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8013149Z outputs = layer_module( 2025-12-04T09:48:03.8013471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8013541Z outputs = self.rel_attn( 2025-12-04T09:48:03.8013799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.8013898Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.8013902Z 2025-12-04T09:48:03.8014001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8014203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8014264Z res = mod(**inputs) 2025-12-04T09:48:03.8014521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8014597Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8014847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8014920Z outputs = layer_module( 2025-12-04T09:48:03.8015169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8015235Z outputs = self.rel_attn( 2025-12-04T09:48:03.8015488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8015557Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8015829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.8015954Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.8015958Z 2025-12-04T09:48:03.8016056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8016257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8016321Z res = mod(**inputs) 2025-12-04T09:48:03.8016593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8016671Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8016935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8017007Z outputs = layer_module( 2025-12-04T09:48:03.8017251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8017315Z outputs = self.rel_attn( 2025-12-04T09:48:03.8017565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.8017691Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.8017694Z 2025-12-04T09:48:03.8017812Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8018002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8018063Z res = mod(**inputs) 2025-12-04T09:48:03.8018318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8018394Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8018645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8018707Z outputs = layer_module( 2025-12-04T09:48:03.8018949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8019040Z outputs = self.rel_attn( 2025-12-04T09:48:03.8019290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8019359Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8019635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.8019758Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.8019761Z 2025-12-04T09:48:03.8019865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8020055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8020114Z res = mod(**inputs) 2025-12-04T09:48:03.8020369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8020445Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8020701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8020766Z outputs = layer_module( 2025-12-04T09:48:03.8021014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8021085Z outputs = self.rel_attn( 2025-12-04T09:48:03.8021333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.8021426Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.8021429Z 2025-12-04T09:48:03.8021533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8021723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8021788Z res = mod(**inputs) 2025-12-04T09:48:03.8022043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8022120Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8022389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8022454Z outputs = layer_module( 2025-12-04T09:48:03.8022709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8022780Z outputs = self.rel_attn( 2025-12-04T09:48:03.8023018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8023091Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8023344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.8023459Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.8023462Z 2025-12-04T09:48:03.8023564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8023761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8023829Z res = mod(**inputs) 2025-12-04T09:48:03.8024068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8024144Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8024389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8024450Z outputs = layer_module( 2025-12-04T09:48:03.8024685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8024753Z outputs = self.rel_attn( 2025-12-04T09:48:03.8025006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8025095Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8025354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8025458Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8025462Z 2025-12-04T09:48:03.8025564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8025751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8025815Z res = mod(**inputs) 2025-12-04T09:48:03.8026059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8026133Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8026379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8026442Z outputs = layer_module( 2025-12-04T09:48:03.8026683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8026750Z outputs = self.rel_attn( 2025-12-04T09:48:03.8026988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8027075Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8027332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8027434Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8027437Z 2025-12-04T09:48:03.8027537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8027723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8027791Z res = mod(**inputs) 2025-12-04T09:48:03.8028050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8028127Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8028392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8028454Z outputs = layer_module( 2025-12-04T09:48:03.8028692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8028891Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8029140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8029222Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8029485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8029553Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8029798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.8029866Z output = self.layer_1(output) 2025-12-04T09:48:03.8029869Z 2025-12-04T09:48:03.8029971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8030156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8030213Z res = mod(**inputs) 2025-12-04T09:48:03.8030455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8030547Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8030790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8030862Z outputs = layer_module( 2025-12-04T09:48:03.8031100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8031298Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8031543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8031615Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8031860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8031926Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8032169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.8032251Z output = self.activation_function(output) 2025-12-04T09:48:03.8032451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.8032523Z return self.act(input) 2025-12-04T09:48:03.8032527Z 2025-12-04T09:48:03.8032623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8032809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8032875Z res = mod(**inputs) 2025-12-04T09:48:03.8033114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8033195Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8033433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8033496Z outputs = layer_module( 2025-12-04T09:48:03.8033755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8033950Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8034221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8034293Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8034536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8034609Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8034850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.8034918Z output = self.layer_2(output) 2025-12-04T09:48:03.8034929Z 2025-12-04T09:48:03.8035041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8035228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8035296Z res = mod(**inputs) 2025-12-04T09:48:03.8035548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8035627Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8035879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8035942Z outputs = layer_module( 2025-12-04T09:48:03.8036190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8036254Z outputs = self.rel_attn( 2025-12-04T09:48:03.8036505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.8036607Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.8036611Z 2025-12-04T09:48:03.8036707Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8036890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8036959Z res = mod(**inputs) 2025-12-04T09:48:03.8037197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8037278Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8037515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8037578Z outputs = layer_module( 2025-12-04T09:48:03.8037824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8037888Z outputs = self.rel_attn( 2025-12-04T09:48:03.8038124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.8038225Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.8038230Z 2025-12-04T09:48:03.8038325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8038515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8038573Z res = mod(**inputs) 2025-12-04T09:48:03.8038810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8038891Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8039128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8039197Z outputs = layer_module( 2025-12-04T09:48:03.8039452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8039516Z outputs = self.rel_attn( 2025-12-04T09:48:03.8039758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8039841Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8040103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.8040232Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.8040236Z 2025-12-04T09:48:03.8040332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8040529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8040588Z res = mod(**inputs) 2025-12-04T09:48:03.8040842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8040923Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8041156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8041224Z outputs = layer_module( 2025-12-04T09:48:03.8041454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8041515Z outputs = self.rel_attn( 2025-12-04T09:48:03.8041750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.8041866Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.8041885Z 2025-12-04T09:48:03.8041979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8042166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8042226Z res = mod(**inputs) 2025-12-04T09:48:03.8042473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8042547Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8042818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8042886Z outputs = layer_module( 2025-12-04T09:48:03.8043125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8043195Z outputs = self.rel_attn( 2025-12-04T09:48:03.8043436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8043506Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8043775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.8043894Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.8043898Z 2025-12-04T09:48:03.8043994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8044187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8044246Z res = mod(**inputs) 2025-12-04T09:48:03.8044502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8044578Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8044820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8044892Z outputs = layer_module( 2025-12-04T09:48:03.8045147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8045220Z outputs = self.rel_attn( 2025-12-04T09:48:03.8045459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.8045571Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.8045574Z 2025-12-04T09:48:03.8045676Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8045865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8045925Z res = mod(**inputs) 2025-12-04T09:48:03.8046183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8046261Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8046534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8046599Z outputs = layer_module( 2025-12-04T09:48:03.8046846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8046920Z outputs = self.rel_attn( 2025-12-04T09:48:03.8047166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8047233Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8047504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.8047623Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.8047642Z 2025-12-04T09:48:03.8047748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8047940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8048001Z res = mod(**inputs) 2025-12-04T09:48:03.8048261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8048340Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8048596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8048659Z outputs = layer_module( 2025-12-04T09:48:03.8048905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8048977Z outputs = self.rel_attn( 2025-12-04T09:48:03.8049227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8049314Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8049591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8049700Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8049705Z 2025-12-04T09:48:03.8049812Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8050003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8050064Z res = mod(**inputs) 2025-12-04T09:48:03.8050321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8050398Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8050653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8050720Z outputs = layer_module( 2025-12-04T09:48:03.8051009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8051085Z outputs = self.rel_attn( 2025-12-04T09:48:03.8051336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8051518Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8051814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8051931Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8051935Z 2025-12-04T09:48:03.8052049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8052270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8052340Z res = mod(**inputs) 2025-12-04T09:48:03.8052649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8052739Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8053023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8053091Z outputs = layer_module( 2025-12-04T09:48:03.8053340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8053552Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8053813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8053908Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8054177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8054250Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8054514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.8054582Z output = self.layer_1(output) 2025-12-04T09:48:03.8054585Z 2025-12-04T09:48:03.8054682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8054879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8054940Z res = mod(**inputs) 2025-12-04T09:48:03.8055193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8055270Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8055516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8055588Z outputs = layer_module( 2025-12-04T09:48:03.8055833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8056029Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8056291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8056364Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8056617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8056684Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8056928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.8057022Z output = self.activation_function(output) 2025-12-04T09:48:03.8057255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.8057331Z return self.act(input) 2025-12-04T09:48:03.8057334Z 2025-12-04T09:48:03.8057433Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8057637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8057706Z res = mod(**inputs) 2025-12-04T09:48:03.8057953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8058030Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8058281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8058347Z outputs = layer_module( 2025-12-04T09:48:03.8058614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8058819Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8059076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8059158Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8059406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8059482Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8059730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.8059797Z output = self.layer_2(output) 2025-12-04T09:48:03.8059817Z 2025-12-04T09:48:03.8059926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8060117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8060178Z res = mod(**inputs) 2025-12-04T09:48:03.8060427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8060506Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8060758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8060822Z outputs = layer_module( 2025-12-04T09:48:03.8061065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8061138Z outputs = self.rel_attn( 2025-12-04T09:48:03.8061380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.8061486Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.8061491Z 2025-12-04T09:48:03.8061590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8061780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8061846Z res = mod(**inputs) 2025-12-04T09:48:03.8062095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8062172Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8062425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8062488Z outputs = layer_module( 2025-12-04T09:48:03.8062738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8062805Z outputs = self.rel_attn( 2025-12-04T09:48:03.8063066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.8063171Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.8063175Z 2025-12-04T09:48:03.8063287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8063483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8063545Z res = mod(**inputs) 2025-12-04T09:48:03.8063790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8063874Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8064116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8064180Z outputs = layer_module( 2025-12-04T09:48:03.8064446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8064515Z outputs = self.rel_attn( 2025-12-04T09:48:03.8064777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8064848Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8065127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.8065262Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.8065265Z 2025-12-04T09:48:03.8065363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8065562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8065640Z res = mod(**inputs) 2025-12-04T09:48:03.8065896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8065981Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8066233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8066299Z outputs = layer_module( 2025-12-04T09:48:03.8066554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8066619Z outputs = self.rel_attn( 2025-12-04T09:48:03.8066877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.8067005Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.8067009Z 2025-12-04T09:48:03.8067110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8067314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8067386Z res = mod(**inputs) 2025-12-04T09:48:03.8067640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8067724Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8067973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8068042Z outputs = layer_module( 2025-12-04T09:48:03.8068290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8068352Z outputs = self.rel_attn( 2025-12-04T09:48:03.8068609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8068679Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8068952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.8069092Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.8069095Z 2025-12-04T09:48:03.8069211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8069407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8069467Z res = mod(**inputs) 2025-12-04T09:48:03.8069715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8069800Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8070049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8070121Z outputs = layer_module( 2025-12-04T09:48:03.8070384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8070450Z outputs = self.rel_attn( 2025-12-04T09:48:03.8070702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.8070800Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.8070804Z 2025-12-04T09:48:03.8070906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8071096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8071158Z res = mod(**inputs) 2025-12-04T09:48:03.8071413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8071491Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8071756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8071828Z outputs = layer_module( 2025-12-04T09:48:03.8072074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8072147Z outputs = self.rel_attn( 2025-12-04T09:48:03.8072534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8072608Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8072885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.8073006Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.8073010Z 2025-12-04T09:48:03.8073116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8073312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8073374Z res = mod(**inputs) 2025-12-04T09:48:03.8073630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8073709Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8073959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8074032Z outputs = layer_module( 2025-12-04T09:48:03.8074276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8074349Z outputs = self.rel_attn( 2025-12-04T09:48:03.8074594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8074680Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8074955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8075111Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8075115Z 2025-12-04T09:48:03.8075222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8075437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8075497Z res = mod(**inputs) 2025-12-04T09:48:03.8075750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8075828Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8076072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8076145Z outputs = layer_module( 2025-12-04T09:48:03.8076415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8076490Z outputs = self.rel_attn( 2025-12-04T09:48:03.8076741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8076829Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8077105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8077214Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8077217Z 2025-12-04T09:48:03.8077327Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8077522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8077607Z res = mod(**inputs) 2025-12-04T09:48:03.8077864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8077943Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8078191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8078264Z outputs = layer_module( 2025-12-04T09:48:03.8078507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8078709Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8078964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8079037Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8079290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8079358Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8079601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.8079674Z output = self.layer_1(output) 2025-12-04T09:48:03.8079679Z 2025-12-04T09:48:03.8079775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8079971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8080030Z res = mod(**inputs) 2025-12-04T09:48:03.8080275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8080359Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8080604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8080677Z outputs = layer_module( 2025-12-04T09:48:03.8080946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8081146Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8081426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8081499Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8081747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8081821Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8082069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.8082161Z output = self.activation_function(output) 2025-12-04T09:48:03.8082386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.8082455Z return self.act(input) 2025-12-04T09:48:03.8082459Z 2025-12-04T09:48:03.8082566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8082758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8082824Z res = mod(**inputs) 2025-12-04T09:48:03.8083067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8083143Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8083399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8083463Z outputs = layer_module( 2025-12-04T09:48:03.8083726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8083931Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8084187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8084266Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8084512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8084579Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8084829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.8084895Z output = self.layer_2(output) 2025-12-04T09:48:03.8084899Z 2025-12-04T09:48:03.8085005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8085194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8085256Z res = mod(**inputs) 2025-12-04T09:48:03.8085510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8085588Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8085837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8085909Z outputs = layer_module( 2025-12-04T09:48:03.8086150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8086222Z outputs = self.rel_attn( 2025-12-04T09:48:03.8086462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.8086558Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.8086562Z 2025-12-04T09:48:03.8086667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8086867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8086935Z res = mod(**inputs) 2025-12-04T09:48:03.8087195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8087274Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8087534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8087596Z outputs = layer_module( 2025-12-04T09:48:03.8087832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8087905Z outputs = self.rel_attn( 2025-12-04T09:48:03.8088142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.8088258Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.8088261Z 2025-12-04T09:48:03.8088358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8088547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8088613Z res = mod(**inputs) 2025-12-04T09:48:03.8088854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8088938Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8089177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8089240Z outputs = layer_module( 2025-12-04T09:48:03.8089498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8089564Z outputs = self.rel_attn( 2025-12-04T09:48:03.8089802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8089876Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8090132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.8090260Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.8090263Z 2025-12-04T09:48:03.8090355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8090537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8090603Z res = mod(**inputs) 2025-12-04T09:48:03.8090842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8090919Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8091167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8091235Z outputs = layer_module( 2025-12-04T09:48:03.8091568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8091642Z outputs = self.rel_attn( 2025-12-04T09:48:03.8091919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.8092067Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.8092071Z 2025-12-04T09:48:03.8092179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8092395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8092460Z res = mod(**inputs) 2025-12-04T09:48:03.8092755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8092845Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8093113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8093177Z outputs = layer_module( 2025-12-04T09:48:03.8093433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8093498Z outputs = self.rel_attn( 2025-12-04T09:48:03.8093751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8093820Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8094088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.8094233Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.8094237Z 2025-12-04T09:48:03.8094335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8094528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8094587Z res = mod(**inputs) 2025-12-04T09:48:03.8094827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8094911Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8095152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8095215Z outputs = layer_module( 2025-12-04T09:48:03.8095484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8095547Z outputs = self.rel_attn( 2025-12-04T09:48:03.8095794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.8095887Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.8095892Z 2025-12-04T09:48:03.8095989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8096186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8096245Z res = mod(**inputs) 2025-12-04T09:48:03.8096495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8096569Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8096810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8096879Z outputs = layer_module( 2025-12-04T09:48:03.8097117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8097178Z outputs = self.rel_attn( 2025-12-04T09:48:03.8097420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8097486Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8097747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.8097862Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.8097865Z 2025-12-04T09:48:03.8097960Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8098152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8098212Z res = mod(**inputs) 2025-12-04T09:48:03.8098475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8098551Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8098791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8098875Z outputs = layer_module( 2025-12-04T09:48:03.8099114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8099176Z outputs = self.rel_attn( 2025-12-04T09:48:03.8099422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8099504Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8099769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8099890Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8099894Z 2025-12-04T09:48:03.8099991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8100185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8100247Z res = mod(**inputs) 2025-12-04T09:48:03.8100487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8100569Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8100808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8100876Z outputs = layer_module( 2025-12-04T09:48:03.8101132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8101195Z outputs = self.rel_attn( 2025-12-04T09:48:03.8101440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8101522Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8101789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8101894Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8101897Z 2025-12-04T09:48:03.8101992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8102183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8102242Z res = mod(**inputs) 2025-12-04T09:48:03.8102479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8102566Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8102805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8102874Z outputs = layer_module( 2025-12-04T09:48:03.8103111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8103303Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8103557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8103628Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8103876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8103945Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8104198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.8104287Z output = self.layer_1(output) 2025-12-04T09:48:03.8104290Z 2025-12-04T09:48:03.8104386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8104582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8104647Z res = mod(**inputs) 2025-12-04T09:48:03.8104880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8104963Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8105197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8105260Z outputs = layer_module( 2025-12-04T09:48:03.8105516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8105706Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8105951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8106022Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8106253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8106326Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8106559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.8106639Z output = self.activation_function(output) 2025-12-04T09:48:03.8106859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.8106923Z return self.act(input) 2025-12-04T09:48:03.8106927Z 2025-12-04T09:48:03.8107028Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8107206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8107266Z res = mod(**inputs) 2025-12-04T09:48:03.8107505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8107577Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8107813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8107873Z outputs = layer_module( 2025-12-04T09:48:03.8108100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8108295Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8108533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8108601Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8108842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8108907Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8109146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.8109210Z output = self.layer_2(output) 2025-12-04T09:48:03.8109213Z 2025-12-04T09:48:03.8109306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8109492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8109551Z res = mod(**inputs) 2025-12-04T09:48:03.8109804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8109879Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8110118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8110205Z outputs = layer_module( 2025-12-04T09:48:03.8110439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8110502Z outputs = self.rel_attn( 2025-12-04T09:48:03.8110745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:48:03.8110834Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:48:03.8110838Z 2025-12-04T09:48:03.8110937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8111130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8111190Z res = mod(**inputs) 2025-12-04T09:48:03.8111432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8111508Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8111749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8111810Z outputs = layer_module( 2025-12-04T09:48:03.8112043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8112112Z outputs = self.rel_attn( 2025-12-04T09:48:03.8112344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:48:03.8112455Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:48:03.8112458Z 2025-12-04T09:48:03.8112558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8112738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8112801Z res = mod(**inputs) 2025-12-04T09:48:03.8113036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8113107Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8113348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8113408Z outputs = layer_module( 2025-12-04T09:48:03.8113644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8113715Z outputs = self.rel_attn( 2025-12-04T09:48:03.8113953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8114025Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8114281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:48:03.8114401Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:48:03.8114404Z 2025-12-04T09:48:03.8114506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8114701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8114763Z res = mod(**inputs) 2025-12-04T09:48:03.8114997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8115072Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8115336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8115400Z outputs = layer_module( 2025-12-04T09:48:03.8115635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8115720Z outputs = self.rel_attn( 2025-12-04T09:48:03.8115957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:48:03.8116083Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:48:03.8116086Z 2025-12-04T09:48:03.8116179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8116359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8116427Z res = mod(**inputs) 2025-12-04T09:48:03.8116692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8116776Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8117010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8117072Z outputs = layer_module( 2025-12-04T09:48:03.8117309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8117370Z outputs = self.rel_attn( 2025-12-04T09:48:03.8117607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8117682Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8117938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:48:03.8118083Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:48:03.8118086Z 2025-12-04T09:48:03.8118184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8118368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8118435Z res = mod(**inputs) 2025-12-04T09:48:03.8118675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8118758Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8118995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8119056Z outputs = layer_module( 2025-12-04T09:48:03.8119330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8119392Z outputs = self.rel_attn( 2025-12-04T09:48:03.8119623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:48:03.8119722Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:48:03.8119725Z 2025-12-04T09:48:03.8119819Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8120014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8120072Z res = mod(**inputs) 2025-12-04T09:48:03.8120310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8120391Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8120638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8120707Z outputs = layer_module( 2025-12-04T09:48:03.8120995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8121072Z outputs = self.rel_attn( 2025-12-04T09:48:03.8121319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:48:03.8121401Z attn_vec = self.rel_attn_core( 2025-12-04T09:48:03.8121661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:48:03.8121786Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:48:03.8121789Z 2025-12-04T09:48:03.8121886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8122121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8122182Z res = mod(**inputs) 2025-12-04T09:48:03.8122442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8122528Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8122777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8122850Z outputs = layer_module( 2025-12-04T09:48:03.8123090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8123152Z outputs = self.rel_attn( 2025-12-04T09:48:03.8123401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8123482Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8123740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8123874Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8123879Z 2025-12-04T09:48:03.8123977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8124173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8124233Z res = mod(**inputs) 2025-12-04T09:48:03.8124474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8124556Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8124795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8124856Z outputs = layer_module( 2025-12-04T09:48:03.8125102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:48:03.8125167Z outputs = self.rel_attn( 2025-12-04T09:48:03.8125412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:48:03.8125495Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:48:03.8125750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:48:03.8125862Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:48:03.8125865Z 2025-12-04T09:48:03.8125960Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8126150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8126208Z res = mod(**inputs) 2025-12-04T09:48:03.8126446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8126531Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8126772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8126852Z outputs = layer_module( 2025-12-04T09:48:03.8127096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8127305Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8127560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8127633Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8127874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8127950Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8128201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:48:03.8128274Z output = self.layer_1(output) 2025-12-04T09:48:03.8128278Z 2025-12-04T09:48:03.8128374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8128557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8128625Z res = mod(**inputs) 2025-12-04T09:48:03.8128868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8128943Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8129193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8129257Z outputs = layer_module( 2025-12-04T09:48:03.8129519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8129719Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8129966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8130048Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8130288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8130360Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8130600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:48:03.8130680Z output = self.activation_function(output) 2025-12-04T09:48:03.8130891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:03.8130957Z return self.act(input) 2025-12-04T09:48:03.8130960Z 2025-12-04T09:48:03.8131065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8131253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8131311Z res = mod(**inputs) 2025-12-04T09:48:03.8131626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:48:03.8131708Z transformer_outputs = self.transformer( 2025-12-04T09:48:03.8131952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:48:03.8132025Z outputs = layer_module( 2025-12-04T09:48:03.8132266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:48:03.8132475Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:48:03.8132751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:03.8132828Z return forward_fn(*input_tensors) 2025-12-04T09:48:03.8133094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:48:03.8133182Z output_x = self.ff(output_x) 2025-12-04T09:48:03.8133447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:48:03.8133525Z output = self.layer_2(output) 2025-12-04T09:48:03.8133529Z 2025-12-04T09:48:03.8133642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8133845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8133908Z res = mod(**inputs) 2025-12-04T09:48:03.8134182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1633, in forward 2025-12-04T09:48:03.8134294Z logits = self.lm_loss(transformer_outputs[0]) 2025-12-04T09:48:03.8134298Z 2025-12-04T09:48:03.8134398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:03.8134604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:03.8134666Z res = mod(**inputs) 2025-12-04T09:48:03.8134971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1639, in forward 2025-12-04T09:48:03.8135106Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-12-04T09:48:03.8135109Z 2025-12-04T09:48:17.6324692Z Compilation time (from dynamo_timed): 33.220799739 2025-12-04T09:48:17.6362298Z pass 2025-12-04T09:48:17.6362983Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:17.6363885Z TIMING: _recursive_pre_grad_passes:0.01342 _recursive_joint_graph_passes:1.33963 _recursive_post_grad_passes:0.5983 async_compile.wait:1.056 code_gen:12.84645 inductor_compile:17.93776 backend_compile:27.02619 gc:0.00036 entire_frame_compile:33.2208 total_wall_time:33.2208 2025-12-04T09:48:17.6364904Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:32908 | FakeTensor.__torch_dispatch__:15989 | ProxyTorchDispatchMode.__torch_dispatch__:6774 2025-12-04T09:48:17.6365451Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-12-04T09:48:21.0514753Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:48:21.0515587Z import pynvml # type: ignore[import] 2025-12-04T09:48:24.4796526Z 2025-12-04T09:48:25.5758481Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:48:25.5758831Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:48:25.5782583Z cpu eval YituTechConvBert 2025-12-04T09:48:26.5703714Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:26.8579301Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:27.1558782Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:39.1109845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1110309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1110666Z res = mod(**inputs) 2025-12-04T09:48:39.1111088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1111558Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1112332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1114168Z hidden_states = self.encoder( 2025-12-04T09:48:39.1114774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1115525Z layer_outputs = layer_module( 2025-12-04T09:48:39.1115917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1116326Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1116788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1117261Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1117781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1118230Z self_outputs = self.self( 2025-12-04T09:48:39.1118655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1119117Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1119281Z 2025-12-04T09:48:39.1119404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1119804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1120185Z res = mod(**inputs) 2025-12-04T09:48:39.1120602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1121060Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1121564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1122010Z hidden_states = self.encoder( 2025-12-04T09:48:39.1122459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1122922Z layer_outputs = layer_module( 2025-12-04T09:48:39.1123289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1123684Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1124227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1124677Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1125104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1125537Z self_outputs = self.self( 2025-12-04T09:48:39.1125959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1126400Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1126555Z 2025-12-04T09:48:39.1126670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1127069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1127413Z res = mod(**inputs) 2025-12-04T09:48:39.1127822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1128258Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1128698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1129119Z hidden_states = self.encoder( 2025-12-04T09:48:39.1129569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1130050Z layer_outputs = layer_module( 2025-12-04T09:48:39.1130437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1130895Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1131353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1131907Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1132360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1132802Z self_outputs = self.self( 2025-12-04T09:48:39.1133369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1133851Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1134020Z 2025-12-04T09:48:39.1134110Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1134335Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1134585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1134966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1135308Z res = mod(**inputs) 2025-12-04T09:48:39.1135715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1136140Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1136565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1137024Z hidden_states = self.encoder( 2025-12-04T09:48:39.1137439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1137851Z layer_outputs = layer_module( 2025-12-04T09:48:39.1138215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1138597Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1139018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1139439Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1139868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1140291Z self_outputs = self.self( 2025-12-04T09:48:39.1140688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1141147Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1141319Z 2025-12-04T09:48:39.1141404Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1141655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1142028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1142377Z res = mod(**inputs) 2025-12-04T09:48:39.1142776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1143207Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1143640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1144066Z hidden_states = self.encoder( 2025-12-04T09:48:39.1144483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1144892Z layer_outputs = layer_module( 2025-12-04T09:48:39.1145326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1145715Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1146166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1146597Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1147029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1147451Z self_outputs = self.self( 2025-12-04T09:48:39.1147851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1148370Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1148895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1149308Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1149447Z 2025-12-04T09:48:39.1149554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1149924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1150259Z res = mod(**inputs) 2025-12-04T09:48:39.1150694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1151112Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1151548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1151993Z hidden_states = self.encoder( 2025-12-04T09:48:39.1152382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1152779Z layer_outputs = layer_module( 2025-12-04T09:48:39.1153148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1153536Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1153953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1154379Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1154809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1155225Z self_outputs = self.self( 2025-12-04T09:48:39.1155629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1156135Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1156779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1157215Z x = self.pointwise(x) 2025-12-04T09:48:39.1157342Z 2025-12-04T09:48:39.1157452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1157832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1158177Z res = mod(**inputs) 2025-12-04T09:48:39.1158566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1159009Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1159450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1159912Z hidden_states = self.encoder( 2025-12-04T09:48:39.1160357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1160815Z layer_outputs = layer_module( 2025-12-04T09:48:39.1161193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1161579Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1162012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1162456Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1162891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1163319Z self_outputs = self.self( 2025-12-04T09:48:39.1163767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1164275Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1164498Z 2025-12-04T09:48:39.1164613Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1164984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1165323Z res = mod(**inputs) 2025-12-04T09:48:39.1165729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1166179Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1166620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1167109Z hidden_states = self.encoder( 2025-12-04T09:48:39.1167550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1167978Z layer_outputs = layer_module( 2025-12-04T09:48:39.1168364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1168761Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1169192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1169641Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1170088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1170523Z self_outputs = self.self( 2025-12-04T09:48:39.1170941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1171435Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1171804Z 2025-12-04T09:48:39.1171928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1172504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1172873Z res = mod(**inputs) 2025-12-04T09:48:39.1173291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1173737Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1174169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1174606Z hidden_states = self.encoder( 2025-12-04T09:48:39.1175043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1175478Z layer_outputs = layer_module( 2025-12-04T09:48:39.1175928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1176321Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1176796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1177244Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1177685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1178111Z self_outputs = self.self( 2025-12-04T09:48:39.1178531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1179014Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1179215Z 2025-12-04T09:48:39.1179327Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1179554Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1179804Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1180178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1180535Z res = mod(**inputs) 2025-12-04T09:48:39.1180947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1181397Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1181825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1182252Z hidden_states = self.encoder( 2025-12-04T09:48:39.1182713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1183145Z layer_outputs = layer_module( 2025-12-04T09:48:39.1183518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1183912Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1184358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1184795Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1185238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1185670Z self_outputs = self.self( 2025-12-04T09:48:39.1186090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1186580Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1186767Z 2025-12-04T09:48:39.1186888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1187278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1187629Z res = mod(**inputs) 2025-12-04T09:48:39.1188030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1188480Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1188926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1189360Z hidden_states = self.encoder( 2025-12-04T09:48:39.1189787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1190236Z layer_outputs = layer_module( 2025-12-04T09:48:39.1190617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1191048Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1191502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1191970Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1192415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1192918Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1193413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1193866Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1194021Z 2025-12-04T09:48:39.1194135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1194547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1194909Z res = mod(**inputs) 2025-12-04T09:48:39.1195310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1195758Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1196198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1196660Z hidden_states = self.encoder( 2025-12-04T09:48:39.1197086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1197530Z layer_outputs = layer_module( 2025-12-04T09:48:39.1197940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1198338Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1198785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1199244Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1199692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1200140Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1200622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1201136Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1201613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1202063Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1202216Z 2025-12-04T09:48:39.1202328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1202712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1203061Z res = mod(**inputs) 2025-12-04T09:48:39.1203468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1203905Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1204345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1204764Z hidden_states = self.encoder( 2025-12-04T09:48:39.1205194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1205615Z layer_outputs = layer_module( 2025-12-04T09:48:39.1205996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1206406Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1206834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1207297Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1207729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1208146Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1208609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1209126Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1209604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1210105Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1210517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1210881Z return self.act(input) 2025-12-04T09:48:39.1210998Z 2025-12-04T09:48:39.1211105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1211562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1211940Z res = mod(**inputs) 2025-12-04T09:48:39.1212345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1212801Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1213274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1213709Z hidden_states = self.encoder( 2025-12-04T09:48:39.1214122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1214547Z layer_outputs = layer_module( 2025-12-04T09:48:39.1214917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1215303Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1215724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1216140Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1216539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1216929Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1217364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1217857Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1218316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1218722Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1218867Z 2025-12-04T09:48:39.1218969Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1219327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1219656Z res = mod(**inputs) 2025-12-04T09:48:39.1220023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1220437Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1220872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1221273Z hidden_states = self.encoder( 2025-12-04T09:48:39.1221666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1222086Z layer_outputs = layer_module( 2025-12-04T09:48:39.1222433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1222784Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1223185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1223594Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1223998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1224423Z self_outputs = self.self( 2025-12-04T09:48:39.1224821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1225244Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1225392Z 2025-12-04T09:48:39.1225496Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1225857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1226195Z res = mod(**inputs) 2025-12-04T09:48:39.1226571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1226976Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1227451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1227880Z hidden_states = self.encoder( 2025-12-04T09:48:39.1228275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1228665Z layer_outputs = layer_module( 2025-12-04T09:48:39.1229014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1229377Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1229767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1230174Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1230575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1230975Z self_outputs = self.self( 2025-12-04T09:48:39.1231329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1231717Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1231845Z 2025-12-04T09:48:39.1231951Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1232282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1232586Z res = mod(**inputs) 2025-12-04T09:48:39.1232939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1233324Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1233699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1234074Z hidden_states = self.encoder( 2025-12-04T09:48:39.1234448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1234824Z layer_outputs = layer_module( 2025-12-04T09:48:39.1235166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1235531Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1235907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1236287Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1236672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1237046Z self_outputs = self.self( 2025-12-04T09:48:39.1237409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1237802Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1237948Z 2025-12-04T09:48:39.1238046Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1238254Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1238480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1238830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1239147Z res = mod(**inputs) 2025-12-04T09:48:39.1239509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1239897Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1240294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1240684Z hidden_states = self.encoder( 2025-12-04T09:48:39.1241074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1241448Z layer_outputs = layer_module( 2025-12-04T09:48:39.1241785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1242134Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1242511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1242901Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1243290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1243669Z self_outputs = self.self( 2025-12-04T09:48:39.1244036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1244449Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1244598Z 2025-12-04T09:48:39.1244685Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1244910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1245252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1245566Z res = mod(**inputs) 2025-12-04T09:48:39.1245924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1246320Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1246716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1247107Z hidden_states = self.encoder( 2025-12-04T09:48:39.1247494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1247884Z layer_outputs = layer_module( 2025-12-04T09:48:39.1248254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1248615Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1249018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1249414Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1249805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1250189Z self_outputs = self.self( 2025-12-04T09:48:39.1250561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1251039Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1251650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1252121Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1252264Z 2025-12-04T09:48:39.1252379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1252784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1253126Z res = mod(**inputs) 2025-12-04T09:48:39.1253524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1253974Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1254400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1255606Z hidden_states = self.encoder( 2025-12-04T09:48:39.1255991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1256396Z layer_outputs = layer_module( 2025-12-04T09:48:39.1256750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1257115Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1257519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1257939Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1258332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1258712Z self_outputs = self.self( 2025-12-04T09:48:39.1259087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1259561Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1260031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1260406Z x = self.pointwise(x) 2025-12-04T09:48:39.1260521Z 2025-12-04T09:48:39.1260622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1260976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1261281Z res = mod(**inputs) 2025-12-04T09:48:39.1261637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1262030Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1262422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1262801Z hidden_states = self.encoder( 2025-12-04T09:48:39.1263214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1263603Z layer_outputs = layer_module( 2025-12-04T09:48:39.1263960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1264307Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1264696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1265089Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1265481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1265858Z self_outputs = self.self( 2025-12-04T09:48:39.1266242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1266700Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1266894Z 2025-12-04T09:48:39.1266993Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1267334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1267638Z res = mod(**inputs) 2025-12-04T09:48:39.1267991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1268371Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1268754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1269153Z hidden_states = self.encoder( 2025-12-04T09:48:39.1269520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1269917Z layer_outputs = layer_module( 2025-12-04T09:48:39.1270245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1270589Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1270958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1271344Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1271728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1272101Z self_outputs = self.self( 2025-12-04T09:48:39.1272632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1273068Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1273231Z 2025-12-04T09:48:39.1273341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1273674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1273983Z res = mod(**inputs) 2025-12-04T09:48:39.1274339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1274727Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1275108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1275492Z hidden_states = self.encoder( 2025-12-04T09:48:39.1275868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1276245Z layer_outputs = layer_module( 2025-12-04T09:48:39.1276617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1276964Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1277343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1277846Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1278230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1278611Z self_outputs = self.self( 2025-12-04T09:48:39.1278978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1279402Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1279577Z 2025-12-04T09:48:39.1279652Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1279873Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1280095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1280437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1280748Z res = mod(**inputs) 2025-12-04T09:48:39.1281106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1281491Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1281879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1282256Z hidden_states = self.encoder( 2025-12-04T09:48:39.1282623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1283021Z layer_outputs = layer_module( 2025-12-04T09:48:39.1283355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1283698Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1284075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1284469Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1284852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1285224Z self_outputs = self.self( 2025-12-04T09:48:39.1285582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1286000Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1286158Z 2025-12-04T09:48:39.1286267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1286607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1286918Z res = mod(**inputs) 2025-12-04T09:48:39.1287280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1287678Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1288072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1288460Z hidden_states = self.encoder( 2025-12-04T09:48:39.1288841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1289229Z layer_outputs = layer_module( 2025-12-04T09:48:39.1289552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1289906Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1290312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1290724Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1291116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1291620Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1292085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1292519Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1292677Z 2025-12-04T09:48:39.1292792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1293171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1293545Z res = mod(**inputs) 2025-12-04T09:48:39.1293917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1294327Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1294734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1295126Z hidden_states = self.encoder( 2025-12-04T09:48:39.1295523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1295916Z layer_outputs = layer_module( 2025-12-04T09:48:39.1296259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1296642Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1297035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1297434Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1297820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1298202Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1298619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1299086Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1299509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1299912Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1300048Z 2025-12-04T09:48:39.1300148Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1300493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1300798Z res = mod(**inputs) 2025-12-04T09:48:39.1301158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1301560Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1301943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1302334Z hidden_states = self.encoder( 2025-12-04T09:48:39.1302717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1303100Z layer_outputs = layer_module( 2025-12-04T09:48:39.1303426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1303776Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1304183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1304606Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1304987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1305371Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1305789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1306243Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1306676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1307121Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1307505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1307840Z return self.act(input) 2025-12-04T09:48:39.1307961Z 2025-12-04T09:48:39.1308062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1308426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1308735Z res = mod(**inputs) 2025-12-04T09:48:39.1309089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1309487Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1309877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1310274Z hidden_states = self.encoder( 2025-12-04T09:48:39.1310647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1311022Z layer_outputs = layer_module( 2025-12-04T09:48:39.1311346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1311691Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1312073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1312516Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1312902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1313287Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1313693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1314156Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1314599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1314998Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1315136Z 2025-12-04T09:48:39.1315236Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1315584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1315888Z res = mod(**inputs) 2025-12-04T09:48:39.1316246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1316640Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1317041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1317443Z hidden_states = self.encoder( 2025-12-04T09:48:39.1317832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1318243Z layer_outputs = layer_module( 2025-12-04T09:48:39.1318575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1318932Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1319322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1319717Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1320103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1320490Z self_outputs = self.self( 2025-12-04T09:48:39.1320888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1321300Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1321453Z 2025-12-04T09:48:39.1321559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1321916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1322244Z res = mod(**inputs) 2025-12-04T09:48:39.1322608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1323018Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1323419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1323838Z hidden_states = self.encoder( 2025-12-04T09:48:39.1324227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1324615Z layer_outputs = layer_module( 2025-12-04T09:48:39.1324949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1325303Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1325705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1326116Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1326519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1326910Z self_outputs = self.self( 2025-12-04T09:48:39.1327300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1327713Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1327849Z 2025-12-04T09:48:39.1327957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1328306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1328631Z res = mod(**inputs) 2025-12-04T09:48:39.1329001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1329403Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1329810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1330203Z hidden_states = self.encoder( 2025-12-04T09:48:39.1330596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1330987Z layer_outputs = layer_module( 2025-12-04T09:48:39.1331370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1331811Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1332282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1332752Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1333189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1333620Z self_outputs = self.self( 2025-12-04T09:48:39.1334037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1334499Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1334667Z 2025-12-04T09:48:39.1334753Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1335000Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1335251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1335632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1335990Z res = mod(**inputs) 2025-12-04T09:48:39.1336440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1336852Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1337256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1337655Z hidden_states = self.encoder( 2025-12-04T09:48:39.1338043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1338465Z layer_outputs = layer_module( 2025-12-04T09:48:39.1338809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1339163Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1339567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1339976Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1340391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1340781Z self_outputs = self.self( 2025-12-04T09:48:39.1341163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1341590Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1341743Z 2025-12-04T09:48:39.1341826Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1342057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1342410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1342731Z res = mod(**inputs) 2025-12-04T09:48:39.1343094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1343507Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1343914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1344313Z hidden_states = self.encoder( 2025-12-04T09:48:39.1344697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1345099Z layer_outputs = layer_module( 2025-12-04T09:48:39.1345444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1345822Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1346240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1346694Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1347110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1347500Z self_outputs = self.self( 2025-12-04T09:48:39.1347887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1348374Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1348879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1349283Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1349421Z 2025-12-04T09:48:39.1349523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1349879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1350195Z res = mod(**inputs) 2025-12-04T09:48:39.1350598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1351096Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1351499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1351890Z hidden_states = self.encoder( 2025-12-04T09:48:39.1352307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1352711Z layer_outputs = layer_module( 2025-12-04T09:48:39.1353056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1353410Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1353816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1354227Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1354623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1355019Z self_outputs = self.self( 2025-12-04T09:48:39.1355407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1355890Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1356373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1356771Z x = self.pointwise(x) 2025-12-04T09:48:39.1356883Z 2025-12-04T09:48:39.1356996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1357354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1357672Z res = mod(**inputs) 2025-12-04T09:48:39.1358048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1358462Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1358859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1359261Z hidden_states = self.encoder( 2025-12-04T09:48:39.1359673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1360072Z layer_outputs = layer_module( 2025-12-04T09:48:39.1360411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1360794Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1361201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1361605Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1362012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1362412Z self_outputs = self.self( 2025-12-04T09:48:39.1362805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1363300Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1363525Z 2025-12-04T09:48:39.1363626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1363983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1364304Z res = mod(**inputs) 2025-12-04T09:48:39.1364669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1365079Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1365484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1365875Z hidden_states = self.encoder( 2025-12-04T09:48:39.1366298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1366711Z layer_outputs = layer_module( 2025-12-04T09:48:39.1367077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1367457Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1367892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1368342Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1368788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1369217Z self_outputs = self.self( 2025-12-04T09:48:39.1369642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1370123Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1370305Z 2025-12-04T09:48:39.1370416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1370801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1371144Z res = mod(**inputs) 2025-12-04T09:48:39.1371634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1372091Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1372697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1373150Z hidden_states = self.encoder( 2025-12-04T09:48:39.1373573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1373997Z layer_outputs = layer_module( 2025-12-04T09:48:39.1374371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1374835Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1375265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1375733Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1376160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1376586Z self_outputs = self.self( 2025-12-04T09:48:39.1376988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1377474Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1377667Z 2025-12-04T09:48:39.1377766Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1377996Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1378267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1378647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1378990Z res = mod(**inputs) 2025-12-04T09:48:39.1379385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1379822Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1380255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1380675Z hidden_states = self.encoder( 2025-12-04T09:48:39.1381084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1381554Z layer_outputs = layer_module( 2025-12-04T09:48:39.1381921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1382300Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1382730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1383163Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1383599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1384021Z self_outputs = self.self( 2025-12-04T09:48:39.1384446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1384913Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1385089Z 2025-12-04T09:48:39.1385206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1385577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1385926Z res = mod(**inputs) 2025-12-04T09:48:39.1386299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1386702Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1387106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1387505Z hidden_states = self.encoder( 2025-12-04T09:48:39.1387896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1388293Z layer_outputs = layer_module( 2025-12-04T09:48:39.1388638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1389005Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1389421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1389829Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1390256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1390716Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1391169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1391590Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1391739Z 2025-12-04T09:48:39.1391841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1392205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1392524Z res = mod(**inputs) 2025-12-04T09:48:39.1392954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1393372Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1393775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1394176Z hidden_states = self.encoder( 2025-12-04T09:48:39.1394574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1394977Z layer_outputs = layer_module( 2025-12-04T09:48:39.1395318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1395705Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1396114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1396537Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1396931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1397328Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1397757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1398229Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1398674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1399086Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1399224Z 2025-12-04T09:48:39.1399335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1399686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1400037Z res = mod(**inputs) 2025-12-04T09:48:39.1400445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1400891Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1401324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1401742Z hidden_states = self.encoder( 2025-12-04T09:48:39.1402163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1402586Z layer_outputs = layer_module( 2025-12-04T09:48:39.1402951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1403332Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1403789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1404220Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1404663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1405078Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1405526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1406031Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1406515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1406979Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1407399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1407765Z return self.act(input) 2025-12-04T09:48:39.1407889Z 2025-12-04T09:48:39.1407999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1408379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1408710Z res = mod(**inputs) 2025-12-04T09:48:39.1409120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1409551Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1409987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1410435Z hidden_states = self.encoder( 2025-12-04T09:48:39.1410864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1411288Z layer_outputs = layer_module( 2025-12-04T09:48:39.1411714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1412116Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1412563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1413027Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1413468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1413886Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1414348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1414866Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1415364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1415797Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1415942Z 2025-12-04T09:48:39.1416057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1416427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1416764Z res = mod(**inputs) 2025-12-04T09:48:39.1417155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1417587Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1418011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1418436Z hidden_states = self.encoder( 2025-12-04T09:48:39.1418881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1419300Z layer_outputs = layer_module( 2025-12-04T09:48:39.1419688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1420071Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1420496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1420940Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1421364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1421784Z self_outputs = self.self( 2025-12-04T09:48:39.1422214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1422660Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1422823Z 2025-12-04T09:48:39.1422931Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1423307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1423638Z res = mod(**inputs) 2025-12-04T09:48:39.1424047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1424484Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1424922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1425353Z hidden_states = self.encoder( 2025-12-04T09:48:39.1425782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1426212Z layer_outputs = layer_module( 2025-12-04T09:48:39.1426566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1426949Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1427391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1427821Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1428256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1428689Z self_outputs = self.self( 2025-12-04T09:48:39.1429113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1429546Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1429692Z 2025-12-04T09:48:39.1429802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1430181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1430526Z res = mod(**inputs) 2025-12-04T09:48:39.1430935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1431378Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1431819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1432236Z hidden_states = self.encoder( 2025-12-04T09:48:39.1432640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1433059Z layer_outputs = layer_module( 2025-12-04T09:48:39.1433448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1433836Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1434251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1434700Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1435127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1435545Z self_outputs = self.self( 2025-12-04T09:48:39.1435947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1436388Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1436545Z 2025-12-04T09:48:39.1436636Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1436852Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1437579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1437964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1438301Z res = mod(**inputs) 2025-12-04T09:48:39.1438698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1439135Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1439562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1439977Z hidden_states = self.encoder( 2025-12-04T09:48:39.1440396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1440836Z layer_outputs = layer_module( 2025-12-04T09:48:39.1441204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1441580Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1442005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1442435Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1442852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1443273Z self_outputs = self.self( 2025-12-04T09:48:39.1443676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1444133Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1444295Z 2025-12-04T09:48:39.1444377Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1444626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1445006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1445336Z res = mod(**inputs) 2025-12-04T09:48:39.1445725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1446159Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1446597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1447020Z hidden_states = self.encoder( 2025-12-04T09:48:39.1447458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1447889Z layer_outputs = layer_module( 2025-12-04T09:48:39.1448265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1448669Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1449108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1449574Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1450016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1450456Z self_outputs = self.self( 2025-12-04T09:48:39.1450883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1451404Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1452015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1452495Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1452648Z 2025-12-04T09:48:39.1452767Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1453158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1453501Z res = mod(**inputs) 2025-12-04T09:48:39.1453908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1454354Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1454787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1455222Z hidden_states = self.encoder( 2025-12-04T09:48:39.1455647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1456104Z layer_outputs = layer_module( 2025-12-04T09:48:39.1456477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1456873Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1457315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1457764Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1458197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1458627Z self_outputs = self.self( 2025-12-04T09:48:39.1459042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1459564Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1460104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1460533Z x = self.pointwise(x) 2025-12-04T09:48:39.1460651Z 2025-12-04T09:48:39.1460769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1461147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1461494Z res = mod(**inputs) 2025-12-04T09:48:39.1461897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1462337Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1462765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1463197Z hidden_states = self.encoder( 2025-12-04T09:48:39.1463629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1464040Z layer_outputs = layer_module( 2025-12-04T09:48:39.1464389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1464776Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1465186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1465592Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1466003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1466405Z self_outputs = self.self( 2025-12-04T09:48:39.1466783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1467291Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1467524Z 2025-12-04T09:48:39.1467635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1468018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1468354Z res = mod(**inputs) 2025-12-04T09:48:39.1468766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1469216Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1469659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1470087Z hidden_states = self.encoder( 2025-12-04T09:48:39.1470515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1470954Z layer_outputs = layer_module( 2025-12-04T09:48:39.1471315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1471699Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1472134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1472730Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1473136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1473571Z self_outputs = self.self( 2025-12-04T09:48:39.1473981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1474463Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1474645Z 2025-12-04T09:48:39.1474755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1475137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1475477Z res = mod(**inputs) 2025-12-04T09:48:39.1475861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1476298Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1476726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1477148Z hidden_states = self.encoder( 2025-12-04T09:48:39.1477554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1477975Z layer_outputs = layer_module( 2025-12-04T09:48:39.1478339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1478719Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1479191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1479653Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1480079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1480491Z self_outputs = self.self( 2025-12-04T09:48:39.1480900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1481374Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1481566Z 2025-12-04T09:48:39.1481659Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1481875Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1482129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1482535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1482877Z res = mod(**inputs) 2025-12-04T09:48:39.1483278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1483712Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1484138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1484552Z hidden_states = self.encoder( 2025-12-04T09:48:39.1484964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1485415Z layer_outputs = layer_module( 2025-12-04T09:48:39.1485785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1486164Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1486593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1487028Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1487460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1487894Z self_outputs = self.self( 2025-12-04T09:48:39.1488317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1488796Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1488971Z 2025-12-04T09:48:39.1489086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1489469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1489809Z res = mod(**inputs) 2025-12-04T09:48:39.1490200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1490643Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1491088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1491562Z hidden_states = self.encoder( 2025-12-04T09:48:39.1491998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1492421Z layer_outputs = layer_module( 2025-12-04T09:48:39.1492790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1493178Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1493626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1494071Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1494513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1495021Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1495504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1495938Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1496086Z 2025-12-04T09:48:39.1496203Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1496573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1496915Z res = mod(**inputs) 2025-12-04T09:48:39.1497327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1497761Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1498177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1498587Z hidden_states = self.encoder( 2025-12-04T09:48:39.1498966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1499340Z layer_outputs = layer_module( 2025-12-04T09:48:39.1499676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1500025Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1500449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1500847Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1501247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1501640Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1502066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1502530Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1502977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1503381Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1503516Z 2025-12-04T09:48:39.1503621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1503976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1504300Z res = mod(**inputs) 2025-12-04T09:48:39.1504670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1505075Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1505477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1505873Z hidden_states = self.encoder( 2025-12-04T09:48:39.1506265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1506652Z layer_outputs = layer_module( 2025-12-04T09:48:39.1506993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1507355Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1507769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1508174Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1508566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1508962Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1509373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1509833Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1510262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1510688Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1511072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1511409Z return self.act(input) 2025-12-04T09:48:39.1511516Z 2025-12-04T09:48:39.1511625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1511968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1512292Z res = mod(**inputs) 2025-12-04T09:48:39.1512666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1513055Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1513440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1513837Z hidden_states = self.encoder( 2025-12-04T09:48:39.1514234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1514611Z layer_outputs = layer_module( 2025-12-04T09:48:39.1514940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1515287Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1515666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1516048Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1516436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1516813Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1517231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1517696Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1518141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1518543Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1518674Z 2025-12-04T09:48:39.1518779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1519111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1519415Z res = mod(**inputs) 2025-12-04T09:48:39.1519763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1520143Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1520405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1520475Z hidden_states = self.encoder( 2025-12-04T09:48:39.1520752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1520821Z layer_outputs = layer_module( 2025-12-04T09:48:39.1521033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1521131Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1521383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1521458Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1521719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1521786Z self_outputs = self.self( 2025-12-04T09:48:39.1522045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1522151Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1522155Z 2025-12-04T09:48:39.1522258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1522453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1522515Z res = mod(**inputs) 2025-12-04T09:48:39.1522780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1522856Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1523109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1523184Z hidden_states = self.encoder( 2025-12-04T09:48:39.1523462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1523530Z layer_outputs = layer_module( 2025-12-04T09:48:39.1523750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1523821Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1524080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1524155Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1524408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1524482Z self_outputs = self.self( 2025-12-04T09:48:39.1524738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1524824Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1524827Z 2025-12-04T09:48:39.1524929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1525121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1525188Z res = mod(**inputs) 2025-12-04T09:48:39.1525449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1525526Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1525794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1525860Z hidden_states = self.encoder( 2025-12-04T09:48:39.1526135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1526203Z layer_outputs = layer_module( 2025-12-04T09:48:39.1526422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1526521Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1526782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1526882Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1527141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1527207Z self_outputs = self.self( 2025-12-04T09:48:39.1527481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1527570Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1527576Z 2025-12-04T09:48:39.1527653Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1527736Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1527851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1528053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1528114Z res = mod(**inputs) 2025-12-04T09:48:39.1528378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1528463Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1528725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1528793Z hidden_states = self.encoder( 2025-12-04T09:48:39.1529062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1529155Z layer_outputs = layer_module( 2025-12-04T09:48:39.1529379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1529453Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1529712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1529796Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1530101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1530172Z self_outputs = self.self( 2025-12-04T09:48:39.1530430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1530529Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1530532Z 2025-12-04T09:48:39.1530618Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1530716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1530911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1530982Z res = mod(**inputs) 2025-12-04T09:48:39.1531248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1531334Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1531677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1531754Z hidden_states = self.encoder( 2025-12-04T09:48:39.1532077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1532151Z layer_outputs = layer_module( 2025-12-04T09:48:39.1532401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1532493Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1532864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1532957Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1533272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1533345Z self_outputs = self.self( 2025-12-04T09:48:39.1533630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1533791Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1534064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1534142Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1534146Z 2025-12-04T09:48:39.1534270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1534474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1534539Z res = mod(**inputs) 2025-12-04T09:48:39.1534812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1534891Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1535149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1535223Z hidden_states = self.encoder( 2025-12-04T09:48:39.1535479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1535566Z layer_outputs = layer_module( 2025-12-04T09:48:39.1535798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1535873Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1536150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1536227Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1536498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1536572Z self_outputs = self.self( 2025-12-04T09:48:39.1536843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1537003Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1537277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1537346Z x = self.pointwise(x) 2025-12-04T09:48:39.1537351Z 2025-12-04T09:48:39.1537461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1537656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1537719Z res = mod(**inputs) 2025-12-04T09:48:39.1538000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1538077Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1538358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1538429Z hidden_states = self.encoder( 2025-12-04T09:48:39.1538703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1538782Z layer_outputs = layer_module( 2025-12-04T09:48:39.1539022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1539106Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1539383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1539459Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1539724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1539791Z self_outputs = self.self( 2025-12-04T09:48:39.1540048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1540203Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1540207Z 2025-12-04T09:48:39.1540327Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1540528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1540589Z res = mod(**inputs) 2025-12-04T09:48:39.1540852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1540936Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1541194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1541270Z hidden_states = self.encoder( 2025-12-04T09:48:39.1541531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1541617Z layer_outputs = layer_module( 2025-12-04T09:48:39.1541842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1541916Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1542176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1542262Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1542529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1542602Z self_outputs = self.self( 2025-12-04T09:48:39.1542862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1542981Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1542986Z 2025-12-04T09:48:39.1543094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1543285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1543354Z res = mod(**inputs) 2025-12-04T09:48:39.1543617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1543696Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1543961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1544030Z hidden_states = self.encoder( 2025-12-04T09:48:39.1544291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1544368Z layer_outputs = layer_module( 2025-12-04T09:48:39.1544583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1544662Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1544944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1545026Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1545308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1545375Z self_outputs = self.self( 2025-12-04T09:48:39.1545634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1545760Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1545763Z 2025-12-04T09:48:39.1545838Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1545927Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1546023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1546223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1546293Z res = mod(**inputs) 2025-12-04T09:48:39.1546547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1546630Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1546881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1546946Z hidden_states = self.encoder( 2025-12-04T09:48:39.1547203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1547267Z layer_outputs = layer_module( 2025-12-04T09:48:39.1547493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1547572Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1547822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1547906Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1548159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1548221Z self_outputs = self.self( 2025-12-04T09:48:39.1548482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1548587Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1548591Z 2025-12-04T09:48:39.1548693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1548878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1548936Z res = mod(**inputs) 2025-12-04T09:48:39.1549195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1549268Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1549522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1549595Z hidden_states = self.encoder( 2025-12-04T09:48:39.1549857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1549929Z layer_outputs = layer_module( 2025-12-04T09:48:39.1550135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1550208Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1550473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1550567Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1550826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1550967Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1551220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1551304Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1551308Z 2025-12-04T09:48:39.1551403Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1551588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1551657Z res = mod(**inputs) 2025-12-04T09:48:39.1551925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1552011Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1552268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1552339Z hidden_states = self.encoder( 2025-12-04T09:48:39.1552610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1552676Z layer_outputs = layer_module( 2025-12-04T09:48:39.1552902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1552977Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1553244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1553358Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1553606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1553679Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1553970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1554084Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1554344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1554421Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1554425Z 2025-12-04T09:48:39.1554519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1554713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1554772Z res = mod(**inputs) 2025-12-04T09:48:39.1555032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1555107Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1555357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1555438Z hidden_states = self.encoder( 2025-12-04T09:48:39.1555687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1555752Z layer_outputs = layer_module( 2025-12-04T09:48:39.1555968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1556042Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1556301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1556424Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1556671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1556767Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1557052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1557172Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1557433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1557540Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1557758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1557841Z return self.act(input) 2025-12-04T09:48:39.1557845Z 2025-12-04T09:48:39.1557947Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1558146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1558210Z res = mod(**inputs) 2025-12-04T09:48:39.1558481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1558561Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1558823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1558901Z hidden_states = self.encoder( 2025-12-04T09:48:39.1559161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1559258Z layer_outputs = layer_module( 2025-12-04T09:48:39.1559475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1559549Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1559818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1559899Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1560146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1560226Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1560515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1560652Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1560915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1560991Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1560995Z 2025-12-04T09:48:39.1561101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1561290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1561355Z res = mod(**inputs) 2025-12-04T09:48:39.1561615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1561691Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1561955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1562024Z hidden_states = self.encoder( 2025-12-04T09:48:39.1562281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1562372Z layer_outputs = layer_module( 2025-12-04T09:48:39.1562587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1562689Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1562955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1563033Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1563304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1563371Z self_outputs = self.self( 2025-12-04T09:48:39.1563640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1563745Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1563749Z 2025-12-04T09:48:39.1563854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1564059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1564124Z res = mod(**inputs) 2025-12-04T09:48:39.1564391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1564478Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1564744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1564819Z hidden_states = self.encoder( 2025-12-04T09:48:39.1565087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1565175Z layer_outputs = layer_module( 2025-12-04T09:48:39.1565412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1565488Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1565770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1565853Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1566126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1566201Z self_outputs = self.self( 2025-12-04T09:48:39.1566475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1566557Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1566568Z 2025-12-04T09:48:39.1566669Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1566868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1566939Z res = mod(**inputs) 2025-12-04T09:48:39.1567208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1567288Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1567564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1567633Z hidden_states = self.encoder( 2025-12-04T09:48:39.1567915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1567985Z layer_outputs = layer_module( 2025-12-04T09:48:39.1568211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1568297Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1568584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1568680Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1568958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1569026Z self_outputs = self.self( 2025-12-04T09:48:39.1569298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1569389Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1569393Z 2025-12-04T09:48:39.1569471Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1569560Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1569662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1569872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1569944Z res = mod(**inputs) 2025-12-04T09:48:39.1570210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1570297Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1570559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1570628Z hidden_states = self.encoder( 2025-12-04T09:48:39.1570898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1570966Z layer_outputs = layer_module( 2025-12-04T09:48:39.1571215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1571291Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1571643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1571744Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1572040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1572114Z self_outputs = self.self( 2025-12-04T09:48:39.1572570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1572683Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1572688Z 2025-12-04T09:48:39.1572779Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1572892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1573103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1573189Z res = mod(**inputs) 2025-12-04T09:48:39.1573461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1573551Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1573825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1573893Z hidden_states = self.encoder( 2025-12-04T09:48:39.1574162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1574229Z layer_outputs = layer_module( 2025-12-04T09:48:39.1574445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1574531Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1574835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1574923Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1575187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1575280Z self_outputs = self.self( 2025-12-04T09:48:39.1575550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1575709Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1575981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1576057Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1576061Z 2025-12-04T09:48:39.1576191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1576399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1576461Z res = mod(**inputs) 2025-12-04T09:48:39.1576727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1576818Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1577081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1577157Z hidden_states = self.encoder( 2025-12-04T09:48:39.1577421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1577513Z layer_outputs = layer_module( 2025-12-04T09:48:39.1577746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1577823Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1578101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1578180Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1578453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1578529Z self_outputs = self.self( 2025-12-04T09:48:39.1578802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1578958Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1579243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1579314Z x = self.pointwise(x) 2025-12-04T09:48:39.1579317Z 2025-12-04T09:48:39.1579430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1579629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1579691Z res = mod(**inputs) 2025-12-04T09:48:39.1579973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1580051Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1580329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1580398Z hidden_states = self.encoder( 2025-12-04T09:48:39.1580672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1580747Z layer_outputs = layer_module( 2025-12-04T09:48:39.1580992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1581070Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1581341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1581452Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1581725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1581793Z self_outputs = self.self( 2025-12-04T09:48:39.1582056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1582220Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1582224Z 2025-12-04T09:48:39.1582341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1582549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1582613Z res = mod(**inputs) 2025-12-04T09:48:39.1582878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1582966Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1583229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1583298Z hidden_states = self.encoder( 2025-12-04T09:48:39.1583572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1583659Z layer_outputs = layer_module( 2025-12-04T09:48:39.1583890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1583967Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1584238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1584327Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1584593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1584670Z self_outputs = self.self( 2025-12-04T09:48:39.1584940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1585058Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1585062Z 2025-12-04T09:48:39.1585174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1585371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1585435Z res = mod(**inputs) 2025-12-04T09:48:39.1585709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1585790Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1586063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1586132Z hidden_states = self.encoder( 2025-12-04T09:48:39.1586397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1586476Z layer_outputs = layer_module( 2025-12-04T09:48:39.1586697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1586781Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1587069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1587151Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1587423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1587506Z self_outputs = self.self( 2025-12-04T09:48:39.1587769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1587900Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1587904Z 2025-12-04T09:48:39.1587983Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1588065Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1588171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1588380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1588454Z res = mod(**inputs) 2025-12-04T09:48:39.1588721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1588804Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1589077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1589146Z hidden_states = self.encoder( 2025-12-04T09:48:39.1589417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1589486Z layer_outputs = layer_module( 2025-12-04T09:48:39.1589707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1589809Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1590076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1590163Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1590424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1590493Z self_outputs = self.self( 2025-12-04T09:48:39.1590762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1590873Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1590877Z 2025-12-04T09:48:39.1590978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1591180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1591244Z res = mod(**inputs) 2025-12-04T09:48:39.1591517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1591597Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1591861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1591939Z hidden_states = self.encoder( 2025-12-04T09:48:39.1592200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1592275Z layer_outputs = layer_module( 2025-12-04T09:48:39.1592492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1592568Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1592838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1592919Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1593198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1593348Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1593610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1593698Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1593701Z 2025-12-04T09:48:39.1593799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1593990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1594059Z res = mod(**inputs) 2025-12-04T09:48:39.1594323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1594420Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1594686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1594756Z hidden_states = self.encoder( 2025-12-04T09:48:39.1595026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1595093Z layer_outputs = layer_module( 2025-12-04T09:48:39.1595310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1595390Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1595650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1595758Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1596020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1596095Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1596401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1596523Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1596797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1596878Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1596881Z 2025-12-04T09:48:39.1596981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1597190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1597253Z res = mod(**inputs) 2025-12-04T09:48:39.1597520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1597608Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1597883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1597963Z hidden_states = self.encoder( 2025-12-04T09:48:39.1598222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1598291Z layer_outputs = layer_module( 2025-12-04T09:48:39.1598518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1598595Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1598866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1598952Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1599233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1599372Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1599670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1599789Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1600065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1600177Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1600404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1600471Z return self.act(input) 2025-12-04T09:48:39.1600475Z 2025-12-04T09:48:39.1600588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1600794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1600857Z res = mod(**inputs) 2025-12-04T09:48:39.1601121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1601201Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1601455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1601530Z hidden_states = self.encoder( 2025-12-04T09:48:39.1601785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1601873Z layer_outputs = layer_module( 2025-12-04T09:48:39.1602107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1602182Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1602457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1602539Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1602798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1602877Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1603174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1603309Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1603577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1603656Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1603660Z 2025-12-04T09:48:39.1603767Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1603966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1604026Z res = mod(**inputs) 2025-12-04T09:48:39.1604299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1604377Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1604653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1604722Z hidden_states = self.encoder( 2025-12-04T09:48:39.1604991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1605070Z layer_outputs = layer_module( 2025-12-04T09:48:39.1605317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1605417Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1605684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1605765Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1606037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1606104Z self_outputs = self.self( 2025-12-04T09:48:39.1606368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1606467Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1606471Z 2025-12-04T09:48:39.1606588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1606793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1606857Z res = mod(**inputs) 2025-12-04T09:48:39.1607123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1607211Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1607475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1607550Z hidden_states = self.encoder( 2025-12-04T09:48:39.1607816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1607906Z layer_outputs = layer_module( 2025-12-04T09:48:39.1608149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1608229Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1608511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1608602Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1608881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1608958Z self_outputs = self.self( 2025-12-04T09:48:39.1609255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1609341Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1609347Z 2025-12-04T09:48:39.1609462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1609670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1609744Z res = mod(**inputs) 2025-12-04T09:48:39.1610027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1610111Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1610400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1610472Z hidden_states = self.encoder( 2025-12-04T09:48:39.1610751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1610829Z layer_outputs = layer_module( 2025-12-04T09:48:39.1611063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1611149Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1611450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1611613Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1611932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1612008Z self_outputs = self.self( 2025-12-04T09:48:39.1612310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1612406Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1612410Z 2025-12-04T09:48:39.1612492Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1612581Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1612692Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1612918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1612998Z res = mod(**inputs) 2025-12-04T09:48:39.1613277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1613369Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1613649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1613722Z hidden_states = self.encoder( 2025-12-04T09:48:39.1614020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1614093Z layer_outputs = layer_module( 2025-12-04T09:48:39.1614324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1614431Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1614717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1614808Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1615091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1615162Z self_outputs = self.self( 2025-12-04T09:48:39.1615452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1615561Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1615565Z 2025-12-04T09:48:39.1615653Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1615760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1615972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1616047Z res = mod(**inputs) 2025-12-04T09:48:39.1616329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1616411Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1616702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1616776Z hidden_states = self.encoder( 2025-12-04T09:48:39.1617074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1617146Z layer_outputs = layer_module( 2025-12-04T09:48:39.1617386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1617473Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1617770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1617877Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1618158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1618249Z self_outputs = self.self( 2025-12-04T09:48:39.1618542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1618712Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1618997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1619083Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1619089Z 2025-12-04T09:48:39.1619196Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1619429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1619497Z res = mod(**inputs) 2025-12-04T09:48:39.1619777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1619870Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1620152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1620231Z hidden_states = self.encoder( 2025-12-04T09:48:39.1620511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1620584Z layer_outputs = layer_module( 2025-12-04T09:48:39.1620844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1620926Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1621207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1621302Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1621558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1621630Z self_outputs = self.self( 2025-12-04T09:48:39.1621884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1622033Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1622294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1622363Z x = self.pointwise(x) 2025-12-04T09:48:39.1622367Z 2025-12-04T09:48:39.1622471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1622656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1622716Z res = mod(**inputs) 2025-12-04T09:48:39.1622978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1623054Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1623307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1623381Z hidden_states = self.encoder( 2025-12-04T09:48:39.1623636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1623710Z layer_outputs = layer_module( 2025-12-04T09:48:39.1623922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1624023Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1624291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1624386Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1624650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1624716Z self_outputs = self.self( 2025-12-04T09:48:39.1624969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1625123Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1625128Z 2025-12-04T09:48:39.1625226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1625433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1625500Z res = mod(**inputs) 2025-12-04T09:48:39.1625756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1625842Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1626098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1626166Z hidden_states = self.encoder( 2025-12-04T09:48:39.1626432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1626497Z layer_outputs = layer_module( 2025-12-04T09:48:39.1626742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1626818Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1627080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1627164Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1627426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1627493Z self_outputs = self.self( 2025-12-04T09:48:39.1627754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1627868Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1627872Z 2025-12-04T09:48:39.1627978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1628168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1628228Z res = mod(**inputs) 2025-12-04T09:48:39.1628494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1628569Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1628835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1628904Z hidden_states = self.encoder( 2025-12-04T09:48:39.1629161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1629235Z layer_outputs = layer_module( 2025-12-04T09:48:39.1629450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1629524Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1629789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1629882Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1630149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1630232Z self_outputs = self.self( 2025-12-04T09:48:39.1630488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1630619Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1630622Z 2025-12-04T09:48:39.1630698Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1630779Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1630878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1631068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1631134Z res = mod(**inputs) 2025-12-04T09:48:39.1631408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1631488Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1631759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1631827Z hidden_states = self.encoder( 2025-12-04T09:48:39.1632096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1632164Z layer_outputs = layer_module( 2025-12-04T09:48:39.1632384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1632485Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1632745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1632823Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1633085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1633151Z self_outputs = self.self( 2025-12-04T09:48:39.1633415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1633522Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1633526Z 2025-12-04T09:48:39.1633622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1633818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1633880Z res = mod(**inputs) 2025-12-04T09:48:39.1634143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1634223Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1634481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1634557Z hidden_states = self.encoder( 2025-12-04T09:48:39.1634814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1634882Z layer_outputs = layer_module( 2025-12-04T09:48:39.1635101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1635173Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1635438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1635516Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1635791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1635928Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1636203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1636288Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1636292Z 2025-12-04T09:48:39.1636388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1636576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1636645Z res = mod(**inputs) 2025-12-04T09:48:39.1636901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1636979Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1637257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1637327Z hidden_states = self.encoder( 2025-12-04T09:48:39.1637604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1637670Z layer_outputs = layer_module( 2025-12-04T09:48:39.1637875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1637952Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1638200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1638303Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1638556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1638631Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1638928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1639046Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1639303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1639386Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1639390Z 2025-12-04T09:48:39.1639486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1639679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1639740Z res = mod(**inputs) 2025-12-04T09:48:39.1640001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1640096Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1640346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1640419Z hidden_states = self.encoder( 2025-12-04T09:48:39.1640674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1640741Z layer_outputs = layer_module( 2025-12-04T09:48:39.1640962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1641034Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1641290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1641386Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1641814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1641907Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1642209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1642324Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1642580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1642685Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1642894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1642963Z return self.act(input) 2025-12-04T09:48:39.1642967Z 2025-12-04T09:48:39.1643097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1643301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1643363Z res = mod(**inputs) 2025-12-04T09:48:39.1643627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1643713Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1643974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1644048Z hidden_states = self.encoder( 2025-12-04T09:48:39.1644312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1644407Z layer_outputs = layer_module( 2025-12-04T09:48:39.1644633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1644709Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1644988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1645070Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1645333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1645415Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1645725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1645856Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1646137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1646220Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1646223Z 2025-12-04T09:48:39.1646337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1646535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1646600Z res = mod(**inputs) 2025-12-04T09:48:39.1646878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1646958Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1647233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1647302Z hidden_states = self.encoder( 2025-12-04T09:48:39.1647568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1647648Z layer_outputs = layer_module( 2025-12-04T09:48:39.1647893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1647976Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1648267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1648346Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1648616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1648684Z self_outputs = self.self( 2025-12-04T09:48:39.1648948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1649049Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1649053Z 2025-12-04T09:48:39.1649156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1649375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1649439Z res = mod(**inputs) 2025-12-04T09:48:39.1649706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1649793Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1650060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1650131Z hidden_states = self.encoder( 2025-12-04T09:48:39.1650413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1650499Z layer_outputs = layer_module( 2025-12-04T09:48:39.1650730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1650806Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1651074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1651163Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1651430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1651579Z self_outputs = self.self( 2025-12-04T09:48:39.1651878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1651967Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1651971Z 2025-12-04T09:48:39.1652090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1652306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1652376Z res = mod(**inputs) 2025-12-04T09:48:39.1652679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1652777Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1653070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1653145Z hidden_states = self.encoder( 2025-12-04T09:48:39.1653427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1653510Z layer_outputs = layer_module( 2025-12-04T09:48:39.1653746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1653834Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1654136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1654222Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1654513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1654608Z self_outputs = self.self( 2025-12-04T09:48:39.1654898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1655001Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1655005Z 2025-12-04T09:48:39.1655091Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1655180Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1655289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1655500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1655587Z res = mod(**inputs) 2025-12-04T09:48:39.1655869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1655951Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1656239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1656311Z hidden_states = self.encoder( 2025-12-04T09:48:39.1656608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1656681Z layer_outputs = layer_module( 2025-12-04T09:48:39.1656912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1657017Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1657303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1657394Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1657680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1657752Z self_outputs = self.self( 2025-12-04T09:48:39.1658042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1658148Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1658152Z 2025-12-04T09:48:39.1658232Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1658349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1658556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1658630Z res = mod(**inputs) 2025-12-04T09:48:39.1658920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1659001Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1659291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1659366Z hidden_states = self.encoder( 2025-12-04T09:48:39.1659654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1659726Z layer_outputs = layer_module( 2025-12-04T09:48:39.1659960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1660047Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1660330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1660416Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1660722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1660811Z self_outputs = self.self( 2025-12-04T09:48:39.1661098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1661265Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1661546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1661635Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1661641Z 2025-12-04T09:48:39.1661748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1661981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1662050Z res = mod(**inputs) 2025-12-04T09:48:39.1662336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1662429Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1662711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1662786Z hidden_states = self.encoder( 2025-12-04T09:48:39.1663079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1663153Z layer_outputs = layer_module( 2025-12-04T09:48:39.1663400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1663498Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1663780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1663875Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1664149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1664223Z self_outputs = self.self( 2025-12-04T09:48:39.1664483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1664633Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1664898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1664967Z x = self.pointwise(x) 2025-12-04T09:48:39.1664970Z 2025-12-04T09:48:39.1665067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1665267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1665327Z res = mod(**inputs) 2025-12-04T09:48:39.1665592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1665670Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1665927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1666003Z hidden_states = self.encoder( 2025-12-04T09:48:39.1666268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1666345Z layer_outputs = layer_module( 2025-12-04T09:48:39.1666558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1666632Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1666913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1667007Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1667263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1667336Z self_outputs = self.self( 2025-12-04T09:48:39.1667591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1667746Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1667750Z 2025-12-04T09:48:39.1667849Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1668128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1668258Z res = mod(**inputs) 2025-12-04T09:48:39.1668547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1668911Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1669206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1669304Z hidden_states = self.encoder( 2025-12-04T09:48:39.1669618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1669712Z layer_outputs = layer_module( 2025-12-04T09:48:39.1669963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1670107Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1670412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1670544Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1670826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1670951Z self_outputs = self.self( 2025-12-04T09:48:39.1671219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1671388Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1671392Z 2025-12-04T09:48:39.1671554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1671765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1720024Z res = mod(**inputs) 2025-12-04T09:48:39.1720560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1720655Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1720942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1721023Z hidden_states = self.encoder( 2025-12-04T09:48:39.1721287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1721364Z layer_outputs = layer_module( 2025-12-04T09:48:39.1721590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1721678Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1721953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1722038Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1722474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1722603Z self_outputs = self.self( 2025-12-04T09:48:39.1722875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1723053Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1723063Z 2025-12-04T09:48:39.1723153Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1723256Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1723368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1723597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1723696Z res = mod(**inputs) 2025-12-04T09:48:39.1724086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1724179Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1724448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1724520Z hidden_states = self.encoder( 2025-12-04T09:48:39.1724788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1724855Z layer_outputs = layer_module( 2025-12-04T09:48:39.1725082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1725161Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1725485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1725573Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1725866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1725939Z self_outputs = self.self( 2025-12-04T09:48:39.1726222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1726337Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1726342Z 2025-12-04T09:48:39.1726453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1726666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1726730Z res = mod(**inputs) 2025-12-04T09:48:39.1727021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1727105Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1727433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1727508Z hidden_states = self.encoder( 2025-12-04T09:48:39.1727807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1727881Z layer_outputs = layer_module( 2025-12-04T09:48:39.1728118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1728213Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1728507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1728602Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1728915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1729059Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1729354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1729465Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1729469Z 2025-12-04T09:48:39.1729589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1729804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1729871Z res = mod(**inputs) 2025-12-04T09:48:39.1730174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1730263Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1730581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1730665Z hidden_states = self.encoder( 2025-12-04T09:48:39.1730957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1731041Z layer_outputs = layer_module( 2025-12-04T09:48:39.1731281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1731365Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1731754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1731851Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1732161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1732257Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1732588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1732734Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1733028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1733113Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1733118Z 2025-12-04T09:48:39.1733238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1733476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1733555Z res = mod(**inputs) 2025-12-04T09:48:39.1733856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1733943Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1734242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1734317Z hidden_states = self.encoder( 2025-12-04T09:48:39.1734617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1734699Z layer_outputs = layer_module( 2025-12-04T09:48:39.1734931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1735020Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1735320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1735406Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1735711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1735793Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1736121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1736266Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1736546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1736670Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1736898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1736974Z return self.act(input) 2025-12-04T09:48:39.1736985Z 2025-12-04T09:48:39.1737091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1737316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1737393Z res = mod(**inputs) 2025-12-04T09:48:39.1737672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1737759Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1738048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1738120Z hidden_states = self.encoder( 2025-12-04T09:48:39.1738419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1738490Z layer_outputs = layer_module( 2025-12-04T09:48:39.1738739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1738835Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1739093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1739173Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1739430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1739502Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1739801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1739932Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1740189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1740276Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1740281Z 2025-12-04T09:48:39.1740380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1740576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1740640Z res = mod(**inputs) 2025-12-04T09:48:39.1740897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1740980Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1741234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1741310Z hidden_states = self.encoder( 2025-12-04T09:48:39.1741565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1741635Z layer_outputs = layer_module( 2025-12-04T09:48:39.1741872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1741949Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1742204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1742311Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1742571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1742648Z self_outputs = self.self( 2025-12-04T09:48:39.1742906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1742993Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1742998Z 2025-12-04T09:48:39.1743105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1743310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1743376Z res = mod(**inputs) 2025-12-04T09:48:39.1743643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1743722Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1743987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1744055Z hidden_states = self.encoder( 2025-12-04T09:48:39.1744314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1744389Z layer_outputs = layer_module( 2025-12-04T09:48:39.1744622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1744703Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1744962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1745040Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1745308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1745375Z self_outputs = self.self( 2025-12-04T09:48:39.1745632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1745717Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1745721Z 2025-12-04T09:48:39.1745818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1746018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1746078Z res = mod(**inputs) 2025-12-04T09:48:39.1746341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1746427Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1746688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1746761Z hidden_states = self.encoder( 2025-12-04T09:48:39.1747019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1747086Z layer_outputs = layer_module( 2025-12-04T09:48:39.1747306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1747382Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1747643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1747747Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1748009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1748114Z self_outputs = self.self( 2025-12-04T09:48:39.1748372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1748462Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1748465Z 2025-12-04T09:48:39.1748553Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1748629Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1748739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1748929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1748990Z res = mod(**inputs) 2025-12-04T09:48:39.1749270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1749350Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1749614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1749690Z hidden_states = self.encoder( 2025-12-04T09:48:39.1749949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1750024Z layer_outputs = layer_module( 2025-12-04T09:48:39.1750241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1750315Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1750597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1750678Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1750934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1751009Z self_outputs = self.self( 2025-12-04T09:48:39.1751264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1751372Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1751375Z 2025-12-04T09:48:39.1751451Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1751547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1751751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1751813Z res = mod(**inputs) 2025-12-04T09:48:39.1752072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1752148Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1752396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1752469Z hidden_states = self.encoder( 2025-12-04T09:48:39.1752717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1752781Z layer_outputs = layer_module( 2025-12-04T09:48:39.1753001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1753071Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1753327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1753405Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1753668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1753744Z self_outputs = self.self( 2025-12-04T09:48:39.1754014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1754176Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1754428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1754500Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1754503Z 2025-12-04T09:48:39.1754637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1754882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1754967Z res = mod(**inputs) 2025-12-04T09:48:39.1755220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1755295Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1755544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1755607Z hidden_states = self.encoder( 2025-12-04T09:48:39.1755859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1755924Z layer_outputs = layer_module( 2025-12-04T09:48:39.1756137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1756229Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1756484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1756563Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1756821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1756885Z self_outputs = self.self( 2025-12-04T09:48:39.1757142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1757289Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1757545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1757608Z x = self.pointwise(x) 2025-12-04T09:48:39.1757613Z 2025-12-04T09:48:39.1757709Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1757903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1757964Z res = mod(**inputs) 2025-12-04T09:48:39.1758218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1758297Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1758549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1758616Z hidden_states = self.encoder( 2025-12-04T09:48:39.1758893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1758954Z layer_outputs = layer_module( 2025-12-04T09:48:39.1759162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1759229Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1759494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1759568Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1759830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1759894Z self_outputs = self.self( 2025-12-04T09:48:39.1760140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1760280Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1760287Z 2025-12-04T09:48:39.1760379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1760562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1760623Z res = mod(**inputs) 2025-12-04T09:48:39.1760885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1760957Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1761205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1761267Z hidden_states = self.encoder( 2025-12-04T09:48:39.1761516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1761578Z layer_outputs = layer_module( 2025-12-04T09:48:39.1761780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1761867Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1762116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1762187Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1762439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1762500Z self_outputs = self.self( 2025-12-04T09:48:39.1762749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1762857Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1762860Z 2025-12-04T09:48:39.1762953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1763136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1763195Z res = mod(**inputs) 2025-12-04T09:48:39.1763452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1763524Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1763772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1763839Z hidden_states = self.encoder( 2025-12-04T09:48:39.1764086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1764146Z layer_outputs = layer_module( 2025-12-04T09:48:39.1764355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1764422Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1764672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1764744Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1765011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1765078Z self_outputs = self.self( 2025-12-04T09:48:39.1765350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1765469Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1765473Z 2025-12-04T09:48:39.1765542Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1765611Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1765706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1765887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1765947Z res = mod(**inputs) 2025-12-04T09:48:39.1766226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1766303Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1766562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1766627Z hidden_states = self.encoder( 2025-12-04T09:48:39.1766888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1766957Z layer_outputs = layer_module( 2025-12-04T09:48:39.1767164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1767233Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1767493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1767585Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1767851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1767915Z self_outputs = self.self( 2025-12-04T09:48:39.1768175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1768284Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1768287Z 2025-12-04T09:48:39.1768383Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1768576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1768635Z res = mod(**inputs) 2025-12-04T09:48:39.1768891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1768970Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1769231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1769294Z hidden_states = self.encoder( 2025-12-04T09:48:39.1769557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1769620Z layer_outputs = layer_module( 2025-12-04T09:48:39.1769838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1769906Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1770164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1770242Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1770502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1770654Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1770915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1771006Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1771011Z 2025-12-04T09:48:39.1771110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1771300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1771365Z res = mod(**inputs) 2025-12-04T09:48:39.1771751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1771844Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1772165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1772241Z hidden_states = self.encoder( 2025-12-04T09:48:39.1772725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1772811Z layer_outputs = layer_module( 2025-12-04T09:48:39.1773047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1773141Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1773405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1773485Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1773750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1773872Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1774172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1774292Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1774556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1774633Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1774637Z 2025-12-04T09:48:39.1774731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1774917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1774981Z res = mod(**inputs) 2025-12-04T09:48:39.1775232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1775309Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1775564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1775630Z hidden_states = self.encoder( 2025-12-04T09:48:39.1775890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1775954Z layer_outputs = layer_module( 2025-12-04T09:48:39.1776162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1776236Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1776506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1776587Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1776834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1776928Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1777227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1777363Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1777627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1777733Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1777938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1778012Z return self.act(input) 2025-12-04T09:48:39.1778018Z 2025-12-04T09:48:39.1778115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1778331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1778396Z res = mod(**inputs) 2025-12-04T09:48:39.1778657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1778740Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1778996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1779064Z hidden_states = self.encoder( 2025-12-04T09:48:39.1779327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1779393Z layer_outputs = layer_module( 2025-12-04T09:48:39.1779614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1779709Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1779968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1780054Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1780304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1780375Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1780669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1780797Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1781062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1781141Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1781145Z 2025-12-04T09:48:39.1781245Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1781446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1781504Z res = mod(**inputs) 2025-12-04T09:48:39.1781773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1781849Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1782107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1782181Z hidden_states = self.encoder( 2025-12-04T09:48:39.1782439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1782508Z layer_outputs = layer_module( 2025-12-04T09:48:39.1782729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1782824Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1783091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1783184Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1783448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1783523Z self_outputs = self.self( 2025-12-04T09:48:39.1783788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1783884Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1783887Z 2025-12-04T09:48:39.1783988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1784225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1784296Z res = mod(**inputs) 2025-12-04T09:48:39.1784557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1784636Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1784904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1784971Z hidden_states = self.encoder( 2025-12-04T09:48:39.1785238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1785303Z layer_outputs = layer_module( 2025-12-04T09:48:39.1785517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1785616Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1785878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1785963Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1786222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1786291Z self_outputs = self.self( 2025-12-04T09:48:39.1786560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1786637Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1786640Z 2025-12-04T09:48:39.1786739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1786936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1786997Z res = mod(**inputs) 2025-12-04T09:48:39.1787266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1787343Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1787601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1787678Z hidden_states = self.encoder( 2025-12-04T09:48:39.1787938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1788009Z layer_outputs = layer_module( 2025-12-04T09:48:39.1788220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1788293Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1788557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1788635Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1788915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1788990Z self_outputs = self.self( 2025-12-04T09:48:39.1789263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1789357Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1789360Z 2025-12-04T09:48:39.1789435Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1789510Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1789615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1789805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1789868Z res = mod(**inputs) 2025-12-04T09:48:39.1790148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1790231Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1790498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1790567Z hidden_states = self.encoder( 2025-12-04T09:48:39.1790827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1790898Z layer_outputs = layer_module( 2025-12-04T09:48:39.1791111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1791192Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1791481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1791557Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1791835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1791899Z self_outputs = self.self( 2025-12-04T09:48:39.1792149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1792250Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1792254Z 2025-12-04T09:48:39.1792327Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1792428Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1792618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1792677Z res = mod(**inputs) 2025-12-04T09:48:39.1792946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1793023Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1793288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1793356Z hidden_states = self.encoder( 2025-12-04T09:48:39.1793614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1793687Z layer_outputs = layer_module( 2025-12-04T09:48:39.1793897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1793972Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1794237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1794314Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1794594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1794664Z self_outputs = self.self( 2025-12-04T09:48:39.1794918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1795097Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1795354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1795432Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1795436Z 2025-12-04T09:48:39.1795535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1795723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1795791Z res = mod(**inputs) 2025-12-04T09:48:39.1796065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1796144Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1796414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1796485Z hidden_states = self.encoder( 2025-12-04T09:48:39.1796747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1796816Z layer_outputs = layer_module( 2025-12-04T09:48:39.1797031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1797112Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1797380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1797462Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1797713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1797778Z self_outputs = self.self( 2025-12-04T09:48:39.1798038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1798191Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1798456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1798534Z x = self.pointwise(x) 2025-12-04T09:48:39.1798537Z 2025-12-04T09:48:39.1798639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1798841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1798905Z res = mod(**inputs) 2025-12-04T09:48:39.1799172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1799259Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1799521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1799596Z hidden_states = self.encoder( 2025-12-04T09:48:39.1799870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1799935Z layer_outputs = layer_module( 2025-12-04T09:48:39.1800149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1800223Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1800493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1800580Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1800833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1800923Z self_outputs = self.self( 2025-12-04T09:48:39.1801175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1801319Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1801323Z 2025-12-04T09:48:39.1801426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1801608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1801674Z res = mod(**inputs) 2025-12-04T09:48:39.1801939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1802016Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1802277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1802344Z hidden_states = self.encoder( 2025-12-04T09:48:39.1802596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1802667Z layer_outputs = layer_module( 2025-12-04T09:48:39.1802875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1802951Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1803217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1803293Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1803556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1803623Z self_outputs = self.self( 2025-12-04T09:48:39.1803885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1803999Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1804003Z 2025-12-04T09:48:39.1804102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1804300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1804362Z res = mod(**inputs) 2025-12-04T09:48:39.1804622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1804706Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1804963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1805039Z hidden_states = self.encoder( 2025-12-04T09:48:39.1805294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1805360Z layer_outputs = layer_module( 2025-12-04T09:48:39.1805580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1805654Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1805915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1805992Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1806263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1806341Z self_outputs = self.self( 2025-12-04T09:48:39.1806598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1806740Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1806752Z 2025-12-04T09:48:39.1806828Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1806903Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1807016Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1807212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1807281Z res = mod(**inputs) 2025-12-04T09:48:39.1807604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1807707Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1807983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1808054Z hidden_states = self.encoder( 2025-12-04T09:48:39.1808331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1808411Z layer_outputs = layer_module( 2025-12-04T09:48:39.1808643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1808732Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1809010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1809116Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1809409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1809483Z self_outputs = self.self( 2025-12-04T09:48:39.1809761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1809888Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1809892Z 2025-12-04T09:48:39.1810000Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1810213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1810280Z res = mod(**inputs) 2025-12-04T09:48:39.1810562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1810655Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1810939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1811020Z hidden_states = self.encoder( 2025-12-04T09:48:39.1811300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1811375Z layer_outputs = layer_module( 2025-12-04T09:48:39.1811682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1811769Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1812060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1812151Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1812444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1812597Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1812899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1813001Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1813005Z 2025-12-04T09:48:39.1813119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1813330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1813403Z res = mod(**inputs) 2025-12-04T09:48:39.1813684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1813767Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1814059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1814156Z hidden_states = self.encoder( 2025-12-04T09:48:39.1814433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1814502Z layer_outputs = layer_module( 2025-12-04T09:48:39.1814728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1814811Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1815074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1815156Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1815422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1815512Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1815819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1815938Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1816206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1816296Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1816299Z 2025-12-04T09:48:39.1816399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1816601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1816666Z res = mod(**inputs) 2025-12-04T09:48:39.1816932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1817019Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1817286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1817356Z hidden_states = self.encoder( 2025-12-04T09:48:39.1817629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1817699Z layer_outputs = layer_module( 2025-12-04T09:48:39.1817925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1818002Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1818275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1818362Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1818615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1818688Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1819007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1819138Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1819409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1819517Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1819731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1819809Z return self.act(input) 2025-12-04T09:48:39.1819813Z 2025-12-04T09:48:39.1819912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1820116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1820195Z res = mod(**inputs) 2025-12-04T09:48:39.1820464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1820553Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1820820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1820896Z hidden_states = self.encoder( 2025-12-04T09:48:39.1821159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1821228Z layer_outputs = layer_module( 2025-12-04T09:48:39.1821453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1821549Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1821826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1821913Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1822162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1822242Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1822530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1822658Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1822927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1823007Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1823010Z 2025-12-04T09:48:39.1823116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1823312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1823375Z res = mod(**inputs) 2025-12-04T09:48:39.1823654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1823732Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1823990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1824065Z hidden_states = self.encoder( 2025-12-04T09:48:39.1824326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1824401Z layer_outputs = layer_module( 2025-12-04T09:48:39.1824620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1824695Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1824989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1825069Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1825356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1825427Z self_outputs = self.self( 2025-12-04T09:48:39.1825689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1825785Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1825789Z 2025-12-04T09:48:39.1825889Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1826085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1826155Z res = mod(**inputs) 2025-12-04T09:48:39.1826434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1826523Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1826787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1826855Z hidden_states = self.encoder( 2025-12-04T09:48:39.1827125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1827192Z layer_outputs = layer_module( 2025-12-04T09:48:39.1827410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1827510Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1827779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1827866Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1828132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1828201Z self_outputs = self.self( 2025-12-04T09:48:39.1828478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1828558Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1828561Z 2025-12-04T09:48:39.1828668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1828860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1828924Z res = mod(**inputs) 2025-12-04T09:48:39.1829198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1829278Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1829546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1829623Z hidden_states = self.encoder( 2025-12-04T09:48:39.1829891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1829964Z layer_outputs = layer_module( 2025-12-04T09:48:39.1830189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1830265Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1830541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1830620Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1830913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1830983Z self_outputs = self.self( 2025-12-04T09:48:39.1831253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1831369Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1831372Z 2025-12-04T09:48:39.1831451Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1831529Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1831639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1831838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1831912Z res = mod(**inputs) 2025-12-04T09:48:39.1832178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1832273Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1832552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1832623Z hidden_states = self.encoder( 2025-12-04T09:48:39.1832898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1832968Z layer_outputs = layer_module( 2025-12-04T09:48:39.1833189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1833274Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1833542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1833640Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1833921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1833989Z self_outputs = self.self( 2025-12-04T09:48:39.1834265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1834369Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1834373Z 2025-12-04T09:48:39.1834449Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1834557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1834754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1834818Z res = mod(**inputs) 2025-12-04T09:48:39.1835095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1835176Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1835514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1835582Z hidden_states = self.encoder( 2025-12-04T09:48:39.1835853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1835929Z layer_outputs = layer_module( 2025-12-04T09:48:39.1836148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1836233Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1836500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1836581Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1836857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1836938Z self_outputs = self.self( 2025-12-04T09:48:39.1837215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1837405Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1837664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1837744Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1837747Z 2025-12-04T09:48:39.1837847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1838043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1838114Z res = mod(**inputs) 2025-12-04T09:48:39.1838394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1838480Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1838740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1838811Z hidden_states = self.encoder( 2025-12-04T09:48:39.1839076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1839143Z layer_outputs = layer_module( 2025-12-04T09:48:39.1839356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1839436Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1839699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1839806Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1840076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1840144Z self_outputs = self.self( 2025-12-04T09:48:39.1840422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1840576Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1840851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1840920Z x = self.pointwise(x) 2025-12-04T09:48:39.1840923Z 2025-12-04T09:48:39.1841024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1841229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1841292Z res = mod(**inputs) 2025-12-04T09:48:39.1841571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1841654Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1841914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1841985Z hidden_states = self.encoder( 2025-12-04T09:48:39.1842257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1842330Z layer_outputs = layer_module( 2025-12-04T09:48:39.1842545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1842620Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1842889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1842983Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1843250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1843333Z self_outputs = self.self( 2025-12-04T09:48:39.1843596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1843750Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1843753Z 2025-12-04T09:48:39.1843852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1844038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1844109Z res = mod(**inputs) 2025-12-04T09:48:39.1844384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1844473Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1844734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1844802Z hidden_states = self.encoder( 2025-12-04T09:48:39.1845067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1845131Z layer_outputs = layer_module( 2025-12-04T09:48:39.1845355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1845429Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1845687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1845791Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1846053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1846118Z self_outputs = self.self( 2025-12-04T09:48:39.1846390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1846508Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1846511Z 2025-12-04T09:48:39.1846618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1846811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1846874Z res = mod(**inputs) 2025-12-04T09:48:39.1847144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1847224Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1847498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1847567Z hidden_states = self.encoder( 2025-12-04T09:48:39.1847835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1847909Z layer_outputs = layer_module( 2025-12-04T09:48:39.1848128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1848202Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1848478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1848558Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1848832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1848914Z self_outputs = self.self( 2025-12-04T09:48:39.1849184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1849337Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1849340Z 2025-12-04T09:48:39.1849417Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1849500Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1849601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1849794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1849863Z res = mod(**inputs) 2025-12-04T09:48:39.1850128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1850206Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1850496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1850568Z hidden_states = self.encoder( 2025-12-04T09:48:39.1850848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1850917Z layer_outputs = layer_module( 2025-12-04T09:48:39.1851140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1851223Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1851559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1851670Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1851951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1852023Z self_outputs = self.self( 2025-12-04T09:48:39.1852303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1852417Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1852421Z 2025-12-04T09:48:39.1852523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1852729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1852794Z res = mod(**inputs) 2025-12-04T09:48:39.1853069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1853152Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1853425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1853507Z hidden_states = self.encoder( 2025-12-04T09:48:39.1853802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1853876Z layer_outputs = layer_module( 2025-12-04T09:48:39.1854117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1854206Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1854482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1854563Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1854830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1854968Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1855257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1855350Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1855370Z 2025-12-04T09:48:39.1855474Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1855669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1855738Z res = mod(**inputs) 2025-12-04T09:48:39.1856005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1856084Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1856357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1856427Z hidden_states = self.encoder( 2025-12-04T09:48:39.1856722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1856793Z layer_outputs = layer_module( 2025-12-04T09:48:39.1857011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1857098Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1857362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1857450Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1857707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1857799Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1858112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1858233Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1858502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1858592Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1858596Z 2025-12-04T09:48:39.1858695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1858899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1858962Z res = mod(**inputs) 2025-12-04T09:48:39.1859230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1859319Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1859588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1859667Z hidden_states = self.encoder( 2025-12-04T09:48:39.1859933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1860001Z layer_outputs = layer_module( 2025-12-04T09:48:39.1860231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1860305Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1860572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1860663Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1860921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1861003Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1861333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1861452Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1861749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1861859Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1862078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1862147Z return self.act(input) 2025-12-04T09:48:39.1862151Z 2025-12-04T09:48:39.1862252Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1862462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1862523Z res = mod(**inputs) 2025-12-04T09:48:39.1862812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1862900Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1863169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1863245Z hidden_states = self.encoder( 2025-12-04T09:48:39.1863511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1863578Z layer_outputs = layer_module( 2025-12-04T09:48:39.1863813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1863889Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1864179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1864264Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1864523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1864605Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1864905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1865036Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1865309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1865388Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1865393Z 2025-12-04T09:48:39.1865501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1865701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1865766Z res = mod(**inputs) 2025-12-04T09:48:39.1866041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1866121Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1866398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1866467Z hidden_states = self.encoder( 2025-12-04T09:48:39.1866733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1866817Z layer_outputs = layer_module( 2025-12-04T09:48:39.1867025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1867097Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1867376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1867453Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1867730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1867796Z self_outputs = self.self( 2025-12-04T09:48:39.1868050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:48:39.1868143Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:48:39.1868147Z 2025-12-04T09:48:39.1868246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1868439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1868500Z res = mod(**inputs) 2025-12-04T09:48:39.1868770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1868855Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1869108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1869174Z hidden_states = self.encoder( 2025-12-04T09:48:39.1869432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1869498Z layer_outputs = layer_module( 2025-12-04T09:48:39.1869713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1869785Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1870055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1870140Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1870392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1870466Z self_outputs = self.self( 2025-12-04T09:48:39.1870718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:48:39.1870794Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:48:39.1870797Z 2025-12-04T09:48:39.1870900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1871083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1871143Z res = mod(**inputs) 2025-12-04T09:48:39.1871405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1871480Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1871739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1871804Z hidden_states = self.encoder( 2025-12-04T09:48:39.1872058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1872130Z layer_outputs = layer_module( 2025-12-04T09:48:39.1872462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1872548Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1872803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1872885Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1873194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1873265Z self_outputs = self.self( 2025-12-04T09:48:39.1873522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:48:39.1873645Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:48:39.1873649Z 2025-12-04T09:48:39.1873725Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1873806Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1873906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1874104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1874176Z res = mod(**inputs) 2025-12-04T09:48:39.1874446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1874559Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1874826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1874893Z hidden_states = self.encoder( 2025-12-04T09:48:39.1875152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1875217Z layer_outputs = layer_module( 2025-12-04T09:48:39.1875423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1875502Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1875753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1875862Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1876122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1876191Z self_outputs = self.self( 2025-12-04T09:48:39.1876457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:48:39.1876559Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:48:39.1876562Z 2025-12-04T09:48:39.1876643Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1876741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1876929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1876996Z res = mod(**inputs) 2025-12-04T09:48:39.1877254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1877331Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1877597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1877665Z hidden_states = self.encoder( 2025-12-04T09:48:39.1877929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1877998Z layer_outputs = layer_module( 2025-12-04T09:48:39.1878209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1878287Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1878543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1878619Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1878889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1878958Z self_outputs = self.self( 2025-12-04T09:48:39.1879244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1879446Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1879703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:48:39.1879785Z x = self.depthwise(hidden_states) 2025-12-04T09:48:39.1879788Z 2025-12-04T09:48:39.1879885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1880080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1880142Z res = mod(**inputs) 2025-12-04T09:48:39.1880407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1880508Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1880776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1880847Z hidden_states = self.encoder( 2025-12-04T09:48:39.1881115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1881183Z layer_outputs = layer_module( 2025-12-04T09:48:39.1881409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1881484Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1881749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1881864Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1882135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1882213Z self_outputs = self.self( 2025-12-04T09:48:39.1882483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:48:39.1882639Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:48:39.1882915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:48:39.1882983Z x = self.pointwise(x) 2025-12-04T09:48:39.1882987Z 2025-12-04T09:48:39.1883087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1883290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1883355Z res = mod(**inputs) 2025-12-04T09:48:39.1883629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1883709Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1883976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1884055Z hidden_states = self.encoder( 2025-12-04T09:48:39.1884322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1884396Z layer_outputs = layer_module( 2025-12-04T09:48:39.1884616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1884690Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1884965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1885046Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1885332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1885426Z self_outputs = self.self( 2025-12-04T09:48:39.1885693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:48:39.1885850Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:48:39.1885854Z 2025-12-04T09:48:39.1885955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1886150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1886222Z res = mod(**inputs) 2025-12-04T09:48:39.1886506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1886611Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1886894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1886968Z hidden_states = self.encoder( 2025-12-04T09:48:39.1887253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1887324Z layer_outputs = layer_module( 2025-12-04T09:48:39.1887553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1887641Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1887937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1888050Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1888337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1888409Z self_outputs = self.self( 2025-12-04T09:48:39.1888699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:48:39.1888825Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:48:39.1888829Z 2025-12-04T09:48:39.1888943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1889149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1889215Z res = mod(**inputs) 2025-12-04T09:48:39.1889515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1889600Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1889879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1889959Z hidden_states = self.encoder( 2025-12-04T09:48:39.1890238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1890320Z layer_outputs = layer_module( 2025-12-04T09:48:39.1890551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1890630Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1890919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1891002Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1891288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1891361Z self_outputs = self.self( 2025-12-04T09:48:39.1891725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:48:39.1891879Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:48:39.1891900Z 2025-12-04T09:48:39.1891988Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1892072Z cudagraph partition due to non gpu ops 2025-12-04T09:48:39.1892191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1892406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1892483Z res = mod(**inputs) 2025-12-04T09:48:39.1892773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1892862Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1893178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1893250Z hidden_states = self.encoder( 2025-12-04T09:48:39.1893517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1893596Z layer_outputs = layer_module( 2025-12-04T09:48:39.1893816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1893899Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1894172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1894251Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1894555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:48:39.1894625Z self_outputs = self.self( 2025-12-04T09:48:39.1894902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:48:39.1895013Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:48:39.1895017Z 2025-12-04T09:48:39.1895119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1895324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1895386Z res = mod(**inputs) 2025-12-04T09:48:39.1895650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1895735Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1896002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1896079Z hidden_states = self.encoder( 2025-12-04T09:48:39.1896348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1896416Z layer_outputs = layer_module( 2025-12-04T09:48:39.1896644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1896717Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1896992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:48:39.1897070Z self_attention_outputs = self.attention( 2025-12-04T09:48:39.1897337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:48:39.1897473Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:48:39.1897762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:48:39.1897845Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1897855Z 2025-12-04T09:48:39.1897974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1898173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1898244Z res = mod(**inputs) 2025-12-04T09:48:39.1898521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1898598Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1898865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1898934Z hidden_states = self.encoder( 2025-12-04T09:48:39.1899216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1899288Z layer_outputs = layer_module( 2025-12-04T09:48:39.1899506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1899588Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1899852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1899931Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1900194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1900266Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1900588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1900706Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1900965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:48:39.1901053Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1901056Z 2025-12-04T09:48:39.1901154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1901357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1901418Z res = mod(**inputs) 2025-12-04T09:48:39.1901678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1901762Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1902022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1902100Z hidden_states = self.encoder( 2025-12-04T09:48:39.1902359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1902425Z layer_outputs = layer_module( 2025-12-04T09:48:39.1902650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1902726Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1902988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1903075Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1903328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1903415Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1903729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:48:39.1903845Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:48:39.1904121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:48:39.1904247Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:48:39.1904462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:48:39.1904530Z return self.act(input) 2025-12-04T09:48:39.1904533Z 2025-12-04T09:48:39.1904635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1904833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1904896Z res = mod(**inputs) 2025-12-04T09:48:39.1905199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:48:39.1905286Z generator_hidden_states = self.convbert( 2025-12-04T09:48:39.1905544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:48:39.1905619Z hidden_states = self.encoder( 2025-12-04T09:48:39.1905883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:48:39.1905949Z layer_outputs = layer_module( 2025-12-04T09:48:39.1906169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:48:39.1906243Z return super().__call__(*args, **kwargs) 2025-12-04T09:48:39.1906522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:48:39.1906611Z layer_output = apply_chunking_to_forward( 2025-12-04T09:48:39.1906864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:48:39.1906944Z return forward_fn(*input_tensors) 2025-12-04T09:48:39.1907238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:48:39.1907366Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:48:39.1907634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:48:39.1907712Z hidden_states = self.dense(hidden_states) 2025-12-04T09:48:39.1907715Z 2025-12-04T09:48:39.1907821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1908014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1908078Z res = mod(**inputs) 2025-12-04T09:48:39.1908345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 937, in forward 2025-12-04T09:48:39.1908492Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-12-04T09:48:39.1908759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 875, in forward 2025-12-04T09:48:39.1908857Z hidden_states = self.dense(generator_hidden_states) 2025-12-04T09:48:39.1908861Z 2025-12-04T09:48:39.1908958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1909152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1909213Z res = mod(**inputs) 2025-12-04T09:48:39.1909471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-12-04T09:48:39.1909602Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-12-04T09:48:39.1909623Z 2025-12-04T09:48:39.1909724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:48:39.1909915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:48:39.1909993Z res = mod(**inputs) 2025-12-04T09:48:39.1910256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 944, in forward 2025-12-04T09:48:39.1910426Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:48:39.1910429Z 2025-12-04T09:48:50.9221754Z Compilation time (from dynamo_timed): 23.055038467 2025-12-04T09:48:50.9282732Z pass 2025-12-04T09:48:50.9287346Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:48:50.9292325Z TIMING: _recursive_pre_grad_passes:0.00984 _recursive_joint_graph_passes:0.62825 _recursive_post_grad_passes:0.5002 async_compile.wait:0.69186 code_gen:11.03321 inductor_compile:13.5589 backend_compile:18.54 gc:0.00033 entire_frame_compile:23.05504 total_wall_time:23.05504 2025-12-04T09:48:50.9293367Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:13628 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:3966 2025-12-04T09:48:50.9293877Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-12-04T09:48:52.7418014Z accuracy pass_rate=92.59% 2025-12-04T09:48:52.7420787Z calls_captured gmean=0.00x mean=579.519x 2025-12-04T09:48:52.7421585Z unique_graphs gmean=0.00x mean=1.111x 2025-12-04T09:48:52.7430510Z graph_breaks gmean=0.00x mean=0.222x 2025-12-04T09:48:52.7430987Z unique_graph_breaks gmean=0.00x mean=0.074x 2025-12-04T09:48:52.7431736Z autograd_captures gmean=0.00x mean=0.000x 2025-12-04T09:48:52.7432047Z autograd_compiles gmean=0.00x mean=0.000x 2025-12-04T09:48:52.7432296Z cudagraph_skips gmean=0.00x mean=1.111x 2025-12-04T09:48:52.7432548Z compilation_latency mean=21.157 seconds 2025-12-04T09:48:53.7437170Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/cpu_inductor_huggingface_inference.csv 2025-12-04T09:48:54.0165957Z AlbertForMaskedLM PASS 2025-12-04T09:48:54.0166290Z AllenaiLongformerBase PASS 2025-12-04T09:48:54.0166545Z BartForCausalLM PASS 2025-12-04T09:48:54.0166739Z BertForMaskedLM PASS 2025-12-04T09:48:54.0169238Z BlenderbotForCausalLM XFAIL 2025-12-04T09:48:54.0175053Z DebertaV2ForMaskedLM XFAIL 2025-12-04T09:48:54.0184400Z DistilBertForMaskedLM PASS 2025-12-04T09:48:54.0189778Z DistillGPT2 PASS 2025-12-04T09:48:54.0194110Z ElectraForCausalLM PASS 2025-12-04T09:48:54.0198467Z GPT2ForSequenceClassification PASS 2025-12-04T09:48:54.0198750Z GoogleFnet PASS 2025-12-04T09:48:54.0203519Z LayoutLMForMaskedLM PASS 2025-12-04T09:48:54.0203807Z M2M100ForConditionalGeneration PASS 2025-12-04T09:48:54.0207089Z MBartForCausalLM PASS 2025-12-04T09:48:54.0207377Z MT5ForConditionalGeneration PASS 2025-12-04T09:48:54.0207621Z MegatronBertForCausalLM PASS 2025-12-04T09:48:54.0207845Z MobileBertForMaskedLM PASS 2025-12-04T09:48:54.0212048Z OPTForCausalLM PASS 2025-12-04T09:48:54.0224039Z PLBartForCausalLM PASS 2025-12-04T09:48:54.0229075Z PegasusForCausalLM PASS 2025-12-04T09:48:54.0233214Z RobertaForCausalLM PASS 2025-12-04T09:48:54.0233496Z T5ForConditionalGeneration PASS 2025-12-04T09:48:54.0233743Z T5Small PASS 2025-12-04T09:48:54.0242667Z TrOCRForCausalLM PASS 2025-12-04T09:48:54.0242966Z XGLMForCausalLM PASS 2025-12-04T09:48:54.0243512Z XLNetLMHeadModel PASS 2025-12-04T09:48:54.0248083Z YituTechConvBert PASS 2025-12-04T09:48:54.0755304Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/cpu_inductor_huggingface_inference.csv 2025-12-04T09:48:54.3576613Z AlbertForMaskedLM PASS 2025-12-04T09:48:54.3580983Z AllenaiLongformerBase PASS 2025-12-04T09:48:54.3590788Z BartForCausalLM PASS 2025-12-04T09:48:54.3594999Z BertForMaskedLM PASS 2025-12-04T09:48:54.3595297Z BlenderbotForCausalLM PASS 2025-12-04T09:48:54.3595518Z DebertaV2ForMaskedLM PASS 2025-12-04T09:48:54.3595831Z DistilBertForMaskedLM PASS 2025-12-04T09:48:54.3596272Z DistillGPT2 PASS 2025-12-04T09:48:54.3600074Z ElectraForCausalLM PASS 2025-12-04T09:48:54.3608591Z GPT2ForSequenceClassification PASS 2025-12-04T09:48:54.3608847Z GoogleFnet PASS 2025-12-04T09:48:54.3609070Z LayoutLMForMaskedLM PASS 2025-12-04T09:48:54.3609311Z M2M100ForConditionalGeneration PASS 2025-12-04T09:48:54.3615242Z MBartForCausalLM PASS 2025-12-04T09:48:54.3624055Z MT5ForConditionalGeneration PASS 2025-12-04T09:48:54.3624385Z MegatronBertForCausalLM PASS 2025-12-04T09:48:54.3630136Z MobileBertForMaskedLM PASS 2025-12-04T09:48:54.3630561Z OPTForCausalLM PASS 2025-12-04T09:48:54.3640621Z PLBartForCausalLM PASS 2025-12-04T09:48:54.3640900Z PegasusForCausalLM PASS 2025-12-04T09:48:54.3641333Z RobertaForCausalLM PASS 2025-12-04T09:48:54.3641556Z T5ForConditionalGeneration PASS 2025-12-04T09:48:54.3650500Z T5Small PASS 2025-12-04T09:48:54.3650823Z TrOCRForCausalLM PASS 2025-12-04T09:48:54.3651087Z XGLMForCausalLM PASS_BUT_FLAKY 2025-12-04T09:48:54.3651902Z XLNetLMHeadModel PASS 2025-12-04T09:48:54.3660746Z YituTechConvBert PASS 2025-12-04T09:48:54.4173715Z + sccache_epilogue 2025-12-04T09:48:54.4177806Z + echo '::group::Sccache Compilation Log' 2025-12-04T09:48:54.4183156Z ##[group]Sccache Compilation Log 2025-12-04T09:48:54.4187691Z + echo '=================== sccache compilation log ===================' 2025-12-04T09:48:54.4192400Z =================== sccache compilation log =================== 2025-12-04T09:48:54.4194290Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-12-04T09:48:54.4389667Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-12-04T09:48:54.4395502Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-12-04T09:48:54.4397197Z + sccache --show-stats 2025-12-04T09:48:54.4421275Z Compile requests 276 2025-12-04T09:48:54.4421743Z Compile requests executed 0 2025-12-04T09:48:54.4422126Z Cache hits 0 2025-12-04T09:48:54.4422463Z Cache misses 0 2025-12-04T09:48:54.4422814Z Cache hits rate - 2025-12-04T09:48:54.4423485Z Cache timeouts 0 2025-12-04T09:48:54.4423777Z Cache read errors 0 2025-12-04T09:48:54.4423991Z Forced recaches 0 2025-12-04T09:48:54.4424192Z Cache write errors 0 2025-12-04T09:48:54.4424385Z Cache errors 0 2025-12-04T09:48:54.4424573Z Compilations 0 2025-12-04T09:48:54.4424789Z Compilation failures 0 2025-12-04T09:48:54.4424993Z Non-cacheable compilations 0 2025-12-04T09:48:54.4425193Z Non-cacheable calls 25 2025-12-04T09:48:54.4425633Z Non-compilation calls 251 2025-12-04T09:48:54.4425875Z Unsupported compiler calls 0 2025-12-04T09:48:54.4426089Z Average cache write 0.000 s 2025-12-04T09:48:54.4426367Z Average compiler 0.000 s 2025-12-04T09:48:54.4426574Z Average cache read hit 0.000 s 2025-12-04T09:48:54.4426784Z Failed distributed compilations 0 2025-12-04T09:48:54.4426917Z 2025-12-04T09:48:54.4426988Z Non-cacheable reasons: 2025-12-04T09:48:54.4427171Z -E 25 2025-12-04T09:48:54.4427305Z 2025-12-04T09:48:54.4427473Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T09:48:54.4427764Z Version (client) 0.10.0 2025-12-04T09:48:54.4427965Z + sccache --stop-server 2025-12-04T09:48:54.4441437Z Stopping sccache server... 2025-12-04T09:48:54.4448298Z Compile requests 276 2025-12-04T09:48:54.4448789Z Compile requests executed 0 2025-12-04T09:48:54.4449037Z Cache hits 0 2025-12-04T09:48:54.4449243Z Cache misses 0 2025-12-04T09:48:54.4449484Z Cache hits rate - 2025-12-04T09:48:54.4449686Z Cache timeouts 0 2025-12-04T09:48:54.4449897Z Cache read errors 0 2025-12-04T09:48:54.4450100Z Forced recaches 0 2025-12-04T09:48:54.4450307Z Cache write errors 0 2025-12-04T09:48:54.4450513Z Cache errors 0 2025-12-04T09:48:54.4450715Z Compilations 0 2025-12-04T09:48:54.4450926Z Compilation failures 0 2025-12-04T09:48:54.4451136Z Non-cacheable compilations 0 2025-12-04T09:48:54.4451429Z Non-cacheable calls 25 2025-12-04T09:48:54.4451891Z Non-compilation calls 251 2025-12-04T09:48:54.4452116Z Unsupported compiler calls 0 2025-12-04T09:48:54.4452359Z Average cache write 0.000 s 2025-12-04T09:48:54.4452596Z Average compiler 0.000 s 2025-12-04T09:48:54.4452829Z Average cache read hit 0.000 s 2025-12-04T09:48:54.4453068Z Failed distributed compilations 0 2025-12-04T09:48:54.4453226Z 2025-12-04T09:48:54.4453301Z Non-cacheable reasons: 2025-12-04T09:48:54.4453492Z -E 25 2025-12-04T09:48:54.4453623Z 2025-12-04T09:48:54.4453791Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T09:48:54.4454099Z Version (client) 0.10.0 2025-12-04T09:48:54.4454348Z + echo ::endgroup:: 2025-12-04T09:48:54.4454937Z ##[endgroup] 2025-12-04T09:48:54.4455107Z + cleanup_workspace 2025-12-04T09:48:54.4455430Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-12-04T09:48:54.4455936Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-12-04T09:48:54.4456350Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-12-04T09:48:54.4456652Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-12-04T09:48:54.4457024Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-12-04T09:48:54.4457409Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-12-04T09:48:54.4457706Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-12-04T09:48:54.9045552Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-12-04T09:48:54.9045914Z with: 2025-12-04T09:48:54.9046131Z benchmark-results-dir: test/test-reports 2025-12-04T09:48:54.9046380Z dry-run: false 2025-12-04T09:48:54.9046581Z schema-version: v3 2025-12-04T09:48:54.9047008Z github-token: *** 2025-12-04T09:48:54.9047205Z env: 2025-12-04T09:48:54.9047384Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:48:54.9047610Z HAS_NVIDIA_GPU: false 2025-12-04T09:48:54.9047965Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:48:54.9048334Z ##[endgroup] 2025-12-04T09:48:54.9062800Z ##[group]Run set -eux 2025-12-04T09:48:54.9063018Z set -eux 2025-12-04T09:48:54.9063266Z  2025-12-04T09:48:54.9063426Z if [[ -n "" ]]; then 2025-12-04T09:48:54.9063623Z  source "" 2025-12-04T09:48:54.9063794Z fi 2025-12-04T09:48:54.9064082Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-12-04T09:48:54.9064372Z  2025-12-04T09:48:54.9064528Z DEVICE_NAME="" 2025-12-04T09:48:54.9064717Z DEVICE_TYPE="" 2025-12-04T09:48:54.9064890Z  2025-12-04T09:48:54.9065059Z if command -v nvidia-smi; then 2025-12-04T09:48:54.9065368Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-12-04T09:48:54.9065753Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-12-04T09:48:54.9066106Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-12-04T09:48:54.9066390Z  python3 -mpip install torch==2.7.1 2025-12-04T09:48:54.9066634Z elif command -v rocminfo; then 2025-12-04T09:48:54.9066941Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-12-04T09:48:54.9067289Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-12-04T09:48:54.9067648Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-12-04T09:48:54.9067964Z  # GPU device name coming from rocminfo instead 2025-12-04T09:48:54.9068202Z  DEVICE_NAME=rocm 2025-12-04T09:48:54.9068517Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-12-04T09:48:54.9068888Z fi 2025-12-04T09:48:54.9069042Z  2025-12-04T09:48:54.9069231Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-12-04T09:48:54.9069500Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-12-04T09:48:54.9078064Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:48:54.9078321Z env: 2025-12-04T09:48:54.9078487Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:48:54.9078679Z HAS_NVIDIA_GPU: false 2025-12-04T09:48:54.9078982Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:48:54.9079292Z ##[endgroup] 2025-12-04T09:48:54.9113467Z + [[ -n '' ]] 2025-12-04T09:48:54.9117812Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-12-04T09:48:55.0887636Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:48:55.8761901Z Collecting boto3==1.35.33 2025-12-04T09:48:55.8933904Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-12-04T09:48:56.1229038Z Collecting psutil==7.0.0 2025-12-04T09:48:56.1264116Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-12-04T09:48:56.1517520Z Collecting pynvml==12.0.0 2025-12-04T09:48:56.1558267Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-12-04T09:48:56.1947790Z Collecting s3transfer<0.11.0,>=0.10.0 2025-12-04T09:48:56.1985054Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-12-04T09:48:56.2041538Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-12-04T09:48:57.0413066Z Collecting botocore<1.36.0,>=1.35.33 2025-12-04T09:48:57.0446485Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-12-04T09:48:57.1607478Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-12-04T09:48:57.1642723Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-12-04T09:48:57.1708576Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-12-04T09:48:57.1716764Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-12-04T09:48:57.3030373Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-12-04T09:48:57.4053691Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-12-04T09:48:57.7545393Z Attempting uninstall: nvidia-ml-py 2025-12-04T09:48:57.7545850Z Found existing installation: nvidia-ml-py 11.525.84 2025-12-04T09:48:57.7556011Z Uninstalling nvidia-ml-py-11.525.84: 2025-12-04T09:48:57.7687713Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-12-04T09:48:57.8167334Z Attempting uninstall: psutil 2025-12-04T09:48:57.8167697Z Found existing installation: psutil 5.9.8 2025-12-04T09:48:57.8210732Z Uninstalling psutil-5.9.8: 2025-12-04T09:48:57.8216487Z Successfully uninstalled psutil-5.9.8 2025-12-04T09:48:57.9531241Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-12-04T09:48:58.0609309Z + DEVICE_NAME= 2025-12-04T09:48:58.0611313Z + DEVICE_TYPE= 2025-12-04T09:48:58.0611775Z + command -v nvidia-smi 2025-12-04T09:48:58.0611969Z + command -v rocminfo 2025-12-04T09:48:58.0612139Z + echo DEVICE_NAME= 2025-12-04T09:48:58.0612309Z + echo DEVICE_TYPE= 2025-12-04T09:48:58.0626317Z ##[group]Run set -eux 2025-12-04T09:48:58.0626498Z set -eux 2025-12-04T09:48:58.0626648Z  2025-12-04T09:48:58.0626854Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-12-04T09:48:58.0627072Z  echo "Missing github-token input" 2025-12-04T09:48:58.0627270Z  exit 1 2025-12-04T09:48:58.0627486Z fi 2025-12-04T09:48:58.0633490Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:48:58.0633733Z env: 2025-12-04T09:48:58.0633891Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:48:58.0634101Z HAS_NVIDIA_GPU: false 2025-12-04T09:48:58.0634385Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:48:58.0634675Z DEVICE_NAME: 2025-12-04T09:48:58.0634835Z DEVICE_TYPE: 2025-12-04T09:48:58.0635193Z GITHUB_TOKEN: *** 2025-12-04T09:48:58.0635352Z ##[endgroup] 2025-12-04T09:48:58.0657506Z + [[ -z *** ]] 2025-12-04T09:48:58.0686837Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-12-04T09:48:58.0687124Z with: 2025-12-04T09:48:58.0687399Z github-token: *** 2025-12-04T09:48:58.0687559Z env: 2025-12-04T09:48:58.0687718Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:48:58.0687910Z HAS_NVIDIA_GPU: false 2025-12-04T09:48:58.0688207Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:48:58.0688536Z DEVICE_NAME: 2025-12-04T09:48:58.0688700Z DEVICE_TYPE: 2025-12-04T09:48:58.0688854Z ##[endgroup] 2025-12-04T09:48:58.0699694Z ##[group]Run set -eux 2025-12-04T09:48:58.0699889Z set -eux 2025-12-04T09:48:58.0700052Z  2025-12-04T09:48:58.0700372Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-12-04T09:48:58.0705600Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:48:58.0705850Z env: 2025-12-04T09:48:58.0706009Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:48:58.0706199Z HAS_NVIDIA_GPU: false 2025-12-04T09:48:58.0706615Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:48:58.0706938Z DEVICE_NAME: 2025-12-04T09:48:58.0707101Z DEVICE_TYPE: 2025-12-04T09:48:58.0707561Z GITHUB_TOKEN: *** 2025-12-04T09:48:58.0707747Z ##[endgroup] 2025-12-04T09:48:58.0727494Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 19923066595 i-0cbeb234d1c75d5ac 2025-12-04T09:49:00.4637839Z setting job-id=57118563268 2025-12-04T09:49:00.4638564Z setting job-name=periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:49:00.4737048Z ##[group]Run set -eux 2025-12-04T09:49:00.4737354Z set -eux 2025-12-04T09:49:00.4737513Z  2025-12-04T09:49:00.4737682Z if [[ -n "" ]]; then 2025-12-04T09:49:00.4737881Z  source "" 2025-12-04T09:49:00.4738044Z fi 2025-12-04T09:49:00.4738201Z  2025-12-04T09:49:00.4738485Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-12-04T09:49:00.4738826Z  --schema-version "${SCHEMA_VERSION}" \ 2025-12-04T09:49:00.4739058Z  --repo "${REPO}" \ 2025-12-04T09:49:00.4739267Z  --head-branch "${HEAD_BRANCH}" \ 2025-12-04T09:49:00.4739502Z  --head-sha "${HEAD_SHA}" \ 2025-12-04T09:49:00.4739727Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-12-04T09:49:00.4739968Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-12-04T09:49:00.4740186Z  --job-id "${JOB_ID}" \ 2025-12-04T09:49:00.4740391Z  --job-name "${JOB_NAME}" 2025-12-04T09:49:00.4745004Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:00.4745254Z env: 2025-12-04T09:49:00.4745410Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:00.4745590Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:00.4745878Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:00.4746169Z DEVICE_NAME: 2025-12-04T09:49:00.4746327Z DEVICE_TYPE: 2025-12-04T09:49:00.4746486Z SCHEMA_VERSION: v3 2025-12-04T09:49:00.4746650Z REPO: pytorch/pytorch 2025-12-04T09:49:00.4746829Z HEAD_BRANCH: refs/heads/main 2025-12-04T09:49:00.4747048Z HEAD_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:49:00.4747335Z WORKFLOW_RUN_ID: 19923066595 2025-12-04T09:49:00.4747503Z RUN_ATTEMPT: 1 2025-12-04T09:49:00.4747661Z JOB_ID: 57118563268 2025-12-04T09:49:00.4747988Z JOB_NAME: periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:49:00.4748332Z ##[endgroup] 2025-12-04T09:49:00.4771083Z + [[ -n '' ]] 2025-12-04T09:49:00.4773206Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 --workflow-id 19923066595 --run-attempt 1 --job-id 57118563268 --job-name 'periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)' 2025-12-04T09:49:00.5041034Z ##[group]Run set -eux 2025-12-04T09:49:00.5041249Z set -eux 2025-12-04T09:49:00.5041405Z  2025-12-04T09:49:00.5041566Z if [[ -n "" ]]; then 2025-12-04T09:49:00.5041746Z  source "" 2025-12-04T09:49:00.5041902Z fi 2025-12-04T09:49:00.5042036Z  2025-12-04T09:49:00.5042286Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-12-04T09:49:00.5046761Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:00.5047008Z env: 2025-12-04T09:49:00.5047163Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:00.5047336Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:00.5047616Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:00.5048094Z DEVICE_NAME: 2025-12-04T09:49:00.5048257Z DEVICE_TYPE: 2025-12-04T09:49:00.5048412Z ##[endgroup] 2025-12-04T09:49:00.5070614Z + [[ -n '' ]] 2025-12-04T09:49:00.5071463Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-12-04T09:49:00.5399565Z INFO:root:Fail to import torch to get the device name 2025-12-04T09:49:00.5483342Z ##[group]Run set -eux 2025-12-04T09:49:00.5483537Z set -eux 2025-12-04T09:49:00.5483695Z  2025-12-04T09:49:00.5483870Z # TODO (huydhn): Implement this part 2025-12-04T09:49:00.5484110Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:49:00.5489262Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:00.5489521Z env: 2025-12-04T09:49:00.5489680Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:00.5489870Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:00.5490180Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:00.5490501Z DEVICE_NAME: 2025-12-04T09:49:00.5490659Z DEVICE_TYPE: 2025-12-04T09:49:00.5490820Z ##[endgroup] 2025-12-04T09:49:00.5516480Z + echo 'dependencies={}' 2025-12-04T09:49:00.5550962Z ##[group]Run set -eux 2025-12-04T09:49:00.5551167Z set -eux 2025-12-04T09:49:00.5551335Z  2025-12-04T09:49:00.5551483Z if [[ -n "" ]]; then 2025-12-04T09:49:00.5551671Z  source "" 2025-12-04T09:49:00.5551832Z fi 2025-12-04T09:49:00.5551967Z  2025-12-04T09:49:00.5552148Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-12-04T09:49:00.5552432Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-12-04T09:49:00.5552744Z  # We don't want the job to fail if the directory doesn't exist 2025-12-04T09:49:00.5552984Z  exit 0 2025-12-04T09:49:00.5553131Z fi 2025-12-04T09:49:00.5553273Z  2025-12-04T09:49:00.5553439Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-12-04T09:49:00.5553744Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-12-04T09:49:00.5554091Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-12-04T09:49:00.5554348Z  --metadata "${BENCHMARK_METADATA}" \ 2025-12-04T09:49:00.5554666Z  --runners "${RUNNER_INFO}" \ 2025-12-04T09:49:00.5554892Z  --dependencies "${DEPENDENCIES}" \ 2025-12-04T09:49:00.5555106Z  --dry-run 2025-12-04T09:49:00.5555266Z else 2025-12-04T09:49:00.5555509Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-12-04T09:49:00.5555842Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-12-04T09:49:00.5556097Z  --metadata "${BENCHMARK_METADATA}" \ 2025-12-04T09:49:00.5556320Z  --runners "${RUNNER_INFO}" \ 2025-12-04T09:49:00.5556547Z  --dependencies "${DEPENDENCIES}" 2025-12-04T09:49:00.5556750Z fi 2025-12-04T09:49:00.5561142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:00.5561398Z env: 2025-12-04T09:49:00.5561566Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:00.5561753Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:00.5562058Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:00.5562379Z DEVICE_NAME: 2025-12-04T09:49:00.5562536Z DEVICE_TYPE: 2025-12-04T09:49:00.5562724Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-12-04T09:49:00.5562940Z DRY_RUN: false 2025-12-04T09:49:00.5563823Z BENCHMARK_METADATA: {"timestamp": 1764841740, "schema_version": "v3", "name": "periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32", "workflow_id": 19923066595, "run_attempt": 1, "job_id": 57118563268} 2025-12-04T09:49:00.5565014Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-31-90.ec2.internal"}, "name": "", "type": ""}] 2025-12-04T09:49:00.5565405Z DEPENDENCIES: {} 2025-12-04T09:49:00.5565580Z ##[endgroup] 2025-12-04T09:49:00.5589278Z + [[ -n '' ]] 2025-12-04T09:49:00.5594607Z + [[ ! -d test/test-reports ]] 2025-12-04T09:49:00.5595060Z + [[ false == \t\r\u\e ]] 2025-12-04T09:49:00.5596858Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1764841740, "schema_version": "v3", "name": "periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32", "workflow_id": 19923066595, "run_attempt": 1, "job_id": 57118563268}' --runners '[{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-31-90.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-12-04T09:49:00.6731026Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/19923066595/57118563268/inference_huggingface.json 2025-12-04T09:49:00.7002773Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-12-04T09:49:00.9193368Z ##[group]Run cat test/**/*_toprint.log || true 2025-12-04T09:49:00.9193654Z cat test/**/*_toprint.log || true 2025-12-04T09:49:00.9198427Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:00.9198680Z env: 2025-12-04T09:49:00.9198839Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:00.9199018Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:00.9199302Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:00.9199604Z DEVICE_NAME: 2025-12-04T09:49:00.9199759Z DEVICE_TYPE: 2025-12-04T09:49:00.9199911Z ##[endgroup] 2025-12-04T09:49:00.9274109Z cat: 'test/**/*_toprint.log': No such file or directory 2025-12-04T09:49:00.9294860Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-12-04T09:49:00.9295126Z kill "$MONITOR_SCRIPT_PID" 2025-12-04T09:49:00.9299806Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:00.9300041Z env: 2025-12-04T09:49:00.9300197Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:00.9300378Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:00.9300661Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:00.9300964Z DEVICE_NAME: 2025-12-04T09:49:00.9301121Z DEVICE_TYPE: 2025-12-04T09:49:00.9301296Z MONITOR_SCRIPT_PID: 52506 2025-12-04T09:49:00.9301468Z ##[endgroup] 2025-12-04T09:49:00.9321210Z /home/ec2-user/actions-runner/_work/_temp/9c246beb-e26c-4756-a800-102817e649fc.sh: line 1: kill: (52506) - No such process 2025-12-04T09:49:00.9336189Z ##[error]Process completed with exit code 1. 2025-12-04T09:49:00.9423449Z Prepare all required actions 2025-12-04T09:49:00.9423824Z Getting action download info 2025-12-04T09:49:01.1368064Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T09:49:01.3809053Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-12-04T09:49:01.8754586Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-12-04T09:49:01.8754834Z with: 2025-12-04T09:49:01.8755106Z file-suffix: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268 2025-12-04T09:49:01.8755419Z s3-bucket: gha-artifacts 2025-12-04T09:49:01.8755600Z env: 2025-12-04T09:49:01.8755767Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:01.8755950Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:01.8756247Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:01.8756569Z DEVICE_NAME: 2025-12-04T09:49:01.8756725Z DEVICE_TYPE: 2025-12-04T09:49:01.8756912Z ##[endgroup] 2025-12-04T09:49:01.8773045Z ##[group]Run # Remove any previous test jsons if they exist 2025-12-04T09:49:01.8773368Z # Remove any previous test jsons if they exist 2025-12-04T09:49:01.8773635Z rm -f test-jsons-*.zip 2025-12-04T09:49:01.8773934Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-12-04T09:49:01.8778676Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:01.8778927Z env: 2025-12-04T09:49:01.8779093Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:01.8779325Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:01.8779627Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:01.8780028Z DEVICE_NAME: 2025-12-04T09:49:01.8780193Z DEVICE_TYPE: 2025-12-04T09:49:01.8780448Z FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268 2025-12-04T09:49:01.8780743Z ##[endgroup] 2025-12-04T09:49:01.8968929Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-12-04T09:49:01.8993843Z ##[group]Run # Remove any previous test reports if they exist 2025-12-04T09:49:01.8994161Z # Remove any previous test reports if they exist 2025-12-04T09:49:01.8994423Z rm -f test-reports-*.zip 2025-12-04T09:49:01.8994730Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-12-04T09:49:01.8999328Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:01.8999565Z env: 2025-12-04T09:49:01.8999729Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:01.8999903Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:01.9000186Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:01.9000499Z DEVICE_NAME: 2025-12-04T09:49:01.9000659Z DEVICE_TYPE: 2025-12-04T09:49:01.9000903Z FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268 2025-12-04T09:49:01.9001183Z ##[endgroup] 2025-12-04T09:49:01.9069702Z adding: test/test-reports/inference_huggingface.csv (deflated 63%) 2025-12-04T09:49:01.9070096Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-12-04T09:49:01.9070476Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 65%) 2025-12-04T09:49:01.9124883Z ##[group]Run # Remove any previous usage logs if they exist 2025-12-04T09:49:01.9125196Z # Remove any previous usage logs if they exist 2025-12-04T09:49:01.9125423Z rm -f logs-*.zip 2025-12-04T09:49:01.9125659Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-12-04T09:49:01.9125971Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-12-04T09:49:01.9130487Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:01.9130724Z env: 2025-12-04T09:49:01.9130882Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:01.9131053Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:01.9131659Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:01.9131978Z DEVICE_NAME: 2025-12-04T09:49:01.9132139Z DEVICE_TYPE: 2025-12-04T09:49:01.9132385Z FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268 2025-12-04T09:49:01.9132671Z ##[endgroup] 2025-12-04T09:49:01.9172789Z adding: usage_log.txt (deflated 58%) 2025-12-04T09:49:01.9184496Z 2025-12-04T09:49:01.9184936Z zip error: Nothing to do! (logs-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268.zip) 2025-12-04T09:49:01.9249656Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-12-04T09:49:01.9250024Z # Remove any previous debugging artifacts if they exist 2025-12-04T09:49:01.9250273Z rm -f debug-*.zip 2025-12-04T09:49:01.9250460Z if [ -d 'test/debug' ]; then 2025-12-04T09:49:01.9250686Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-12-04T09:49:01.9250889Z fi 2025-12-04T09:49:01.9255210Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:01.9255440Z env: 2025-12-04T09:49:01.9255584Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:01.9255758Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:01.9256026Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:01.9256315Z DEVICE_NAME: 2025-12-04T09:49:01.9256459Z DEVICE_TYPE: 2025-12-04T09:49:01.9256702Z FILE_SUFFIX: test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268 2025-12-04T09:49:01.9256966Z ##[endgroup] 2025-12-04T09:49:01.9319048Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:49:01.9319251Z with: 2025-12-04T09:49:01.9319408Z s3-bucket: gha-artifacts 2025-12-04T09:49:01.9319682Z s3-prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:49:01.9319891Z retention-days: 14 2025-12-04T09:49:01.9320057Z if-no-files-found: warn 2025-12-04T09:49:01.9320233Z path: test-jsons-*.zip 2025-12-04T09:49:01.9320387Z name: artifact 2025-12-04T09:49:01.9320549Z region: us-east-1 2025-12-04T09:49:01.9320700Z env: 2025-12-04T09:49:01.9320836Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:01.9321007Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:01.9321278Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:01.9321566Z DEVICE_NAME: 2025-12-04T09:49:01.9321707Z DEVICE_TYPE: 2025-12-04T09:49:01.9321852Z ##[endgroup] 2025-12-04T09:49:02.2058162Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:49:02.2062995Z With the provided path, there will be 1 file uploaded 2025-12-04T09:49:02.2067448Z Uploading to s3 prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:49:02.2087412Z Starting upload of test-jsons-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268.zip 2025-12-04T09:49:02.3184002Z Finished upload of test-jsons-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268.zip 2025-12-04T09:49:02.3331000Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:49:02.3331257Z with: 2025-12-04T09:49:02.3331431Z s3-bucket: gha-artifacts 2025-12-04T09:49:02.3331922Z s3-prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:49:02.3332171Z retention-days: 14 2025-12-04T09:49:02.3332390Z if-no-files-found: error 2025-12-04T09:49:02.3332693Z path: test-reports-*.zip 2025-12-04T09:49:02.3332875Z name: artifact 2025-12-04T09:49:02.3333052Z region: us-east-1 2025-12-04T09:49:02.3333239Z env: 2025-12-04T09:49:02.3333388Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:02.3333573Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:02.3333877Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:02.3334189Z DEVICE_NAME: 2025-12-04T09:49:02.3334363Z DEVICE_TYPE: 2025-12-04T09:49:02.3334514Z ##[endgroup] 2025-12-04T09:49:02.6178159Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:49:02.6178510Z With the provided path, there will be 1 file uploaded 2025-12-04T09:49:02.6179215Z Uploading to s3 prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:49:02.6209553Z Starting upload of test-reports-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268.zip 2025-12-04T09:49:02.7270894Z Finished upload of test-reports-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268.zip 2025-12-04T09:49:02.7422336Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:49:02.7422573Z with: 2025-12-04T09:49:02.7422744Z s3-bucket: gha-artifacts 2025-12-04T09:49:02.7422971Z s3-prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:49:02.7423205Z retention-days: 14 2025-12-04T09:49:02.7423389Z if-no-files-found: ignore 2025-12-04T09:49:02.7423584Z path: logs-*.zip 2025-12-04T09:49:02.7423755Z name: artifact 2025-12-04T09:49:02.7423923Z region: us-east-1 2025-12-04T09:49:02.7424088Z env: 2025-12-04T09:49:02.7424235Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:02.7424423Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:02.7424733Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:02.7425049Z DEVICE_NAME: 2025-12-04T09:49:02.7425206Z DEVICE_TYPE: 2025-12-04T09:49:02.7425365Z ##[endgroup] 2025-12-04T09:49:02.9932794Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:49:02.9933272Z With the provided path, there will be 1 file uploaded 2025-12-04T09:49:02.9938042Z Uploading to s3 prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:49:02.9963528Z Starting upload of logs-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268.zip 2025-12-04T09:49:03.1128003Z Finished upload of logs-test-cpu_inductor_huggingface-1-1-linux.8xlarge.amx_57118563268.zip 2025-12-04T09:49:03.1269977Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:49:03.1270280Z with: 2025-12-04T09:49:03.1270432Z s3-bucket: gha-artifacts 2025-12-04T09:49:03.1270627Z s3-prefix: pytorch/pytorch/19923066595/1/artifact 2025-12-04T09:49:03.1270836Z retention-days: 14 2025-12-04T09:49:03.1270993Z if-no-files-found: ignore 2025-12-04T09:49:03.1271424Z path: debug-*.zip 2025-12-04T09:49:03.1271575Z name: artifact 2025-12-04T09:49:03.1271724Z region: us-east-1 2025-12-04T09:49:03.1271863Z env: 2025-12-04T09:49:03.1272001Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:03.1272168Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:03.1272681Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:03.1272975Z DEVICE_NAME: 2025-12-04T09:49:03.1273128Z DEVICE_TYPE: 2025-12-04T09:49:03.1273267Z ##[endgroup] 2025-12-04T09:49:03.3750082Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-12-04T09:49:03.3910351Z ##[group]Run # shellcheck disable=SC2156 2025-12-04T09:49:03.3910618Z # shellcheck disable=SC2156 2025-12-04T09:49:03.3910975Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-12-04T09:49:03.3916464Z shell: /usr/bin/bash -e {0} 2025-12-04T09:49:03.3916662Z env: 2025-12-04T09:49:03.3916829Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:03.3917018Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:03.3917331Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:03.3917655Z DEVICE_NAME: 2025-12-04T09:49:03.3917899Z DEVICE_TYPE: 2025-12-04T09:49:03.3918067Z ##[endgroup] 2025-12-04T09:49:03.5716952Z Prepare all required actions 2025-12-04T09:49:03.5717275Z Getting action download info 2025-12-04T09:49:03.7297360Z Download action repository 'actions/setup-python@v6' (SHA:83679a892e2d95755f2dac6acb0bfd1e9ac5d548) 2025-12-04T09:49:04.1331335Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-12-04T09:49:04.1331746Z with: 2025-12-04T09:49:04.1331928Z job_id: 57118563268 2025-12-04T09:49:04.1332309Z job_name: periodic-dynamo-benchmarks-cpu-test / test (cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-12-04T09:49:04.1332732Z workflow_name: inductor-periodic 2025-12-04T09:49:04.1332965Z workflow_run_id: 19923066595 2025-12-04T09:49:04.1333149Z workflow_attempt: 1 2025-12-04T09:49:04.1333311Z env: 2025-12-04T09:49:04.1333456Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:04.1333633Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:04.1333914Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:04.1334248Z DEVICE_NAME: 2025-12-04T09:49:04.1334399Z DEVICE_TYPE: 2025-12-04T09:49:04.1334549Z ##[endgroup] 2025-12-04T09:49:04.1359656Z ##[group]Run actions/setup-python@v6 2025-12-04T09:49:04.1359871Z with: 2025-12-04T09:49:04.1360034Z python-version: 3.10 2025-12-04T09:49:04.1360206Z check-latest: false 2025-12-04T09:49:04.1360629Z token: *** 2025-12-04T09:49:04.1360825Z update-environment: true 2025-12-04T09:49:04.1361021Z allow-prereleases: false 2025-12-04T09:49:04.1361197Z freethreaded: false 2025-12-04T09:49:04.1361364Z env: 2025-12-04T09:49:04.1361518Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:04.1361689Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:04.1361987Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:04.1362301Z DEVICE_NAME: 2025-12-04T09:49:04.1362455Z DEVICE_TYPE: 2025-12-04T09:49:04.1362608Z ##[endgroup] 2025-12-04T09:49:04.2664853Z ##[group]Installed versions 2025-12-04T09:49:04.2668393Z Version 3.10 was not found in the local cache 2025-12-04T09:49:04.2805092Z (node:72288) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. 2025-12-04T09:49:04.2807567Z (Use `node --trace-deprecation ...` to show where the warning was created) 2025-12-04T09:49:04.6055327Z ##[error]The version '3.10' with architecture 'x64' was not found for this operating system. The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json 2025-12-04T09:49:04.6232824Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-12-04T09:49:04.6233096Z with: 2025-12-04T09:49:04.6233235Z env: 2025-12-04T09:49:04.6233385Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:04.6233564Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:04.6233833Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:04.6234124Z DEVICE_NAME: 2025-12-04T09:49:04.6234286Z DEVICE_TYPE: 2025-12-04T09:49:04.6234432Z ##[endgroup] 2025-12-04T09:49:04.6245602Z ##[group]Run set -eou pipefail 2025-12-04T09:49:04.6245817Z set -eou pipefail 2025-12-04T09:49:04.6245986Z  2025-12-04T09:49:04.6246217Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-12-04T09:49:04.6246493Z for _ in $(seq 1440); do 2025-12-04T09:49:04.6246703Z  # Break if no ssh session exists anymore 2025-12-04T09:49:04.6246917Z  if [ "$(who)" = "" ]; then 2025-12-04T09:49:04.6247125Z  break 2025-12-04T09:49:04.6247277Z  fi 2025-12-04T09:49:04.6247424Z  echo "." 2025-12-04T09:49:04.6247579Z  sleep 5 2025-12-04T09:49:04.6247721Z done 2025-12-04T09:49:04.6252448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:04.6252706Z env: 2025-12-04T09:49:04.6252863Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:04.6253131Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:04.6253433Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:04.6253726Z DEVICE_NAME: 2025-12-04T09:49:04.6253881Z DEVICE_TYPE: 2025-12-04T09:49:04.6254037Z ##[endgroup] 2025-12-04T09:49:04.6282191Z Holding runner for 2 hours until all ssh sessions have logged out 2025-12-04T09:49:04.6356261Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:49:04.6356603Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:49:04.6356874Z # shellcheck disable=SC2046 2025-12-04T09:49:04.6357091Z docker stop $(docker ps -q) || true 2025-12-04T09:49:04.6357307Z # Prune all of the docker images 2025-12-04T09:49:04.6357507Z docker system prune -af 2025-12-04T09:49:04.6361786Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:04.6362024Z env: 2025-12-04T09:49:04.6362166Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:49:04.6362349Z HAS_NVIDIA_GPU: false 2025-12-04T09:49:04.6362623Z DOCKER_CONTAINER_ID: b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:04.6362902Z DEVICE_NAME: 2025-12-04T09:49:04.6363050Z DEVICE_TYPE: 2025-12-04T09:49:04.6363200Z ##[endgroup] 2025-12-04T09:49:15.2559413Z b4276b4c571f 2025-12-04T09:49:15.5722018Z Deleted Containers: 2025-12-04T09:49:15.5727155Z b4276b4c571f16ea28d4de4316c61e214d2435c092681997d360946681f74ca3 2025-12-04T09:49:15.5727414Z 2025-12-04T09:49:22.6899995Z Deleted Images: 2025-12-04T09:49:22.6900819Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:49:22.6901730Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:b178ee928adfcab963ebbc6ea05ea1a0f0c605bb095e9076f010bf92d150869d 2025-12-04T09:49:22.6902349Z deleted: sha256:40bd34edc3497e05e2150804a15313235b6426bcc4b8c19c03c3daf5524be913 2025-12-04T09:49:22.6902806Z deleted: sha256:947006dce32e090f6b67d556f519398b957121651577f10819c7b179a5b6b938 2025-12-04T09:49:22.6903285Z deleted: sha256:dca16630fc5508e008012b760ac178fac5af74cb679706a3d9898100501d6f8e 2025-12-04T09:49:22.6903664Z deleted: sha256:a35e2e4cfaaa05e1f193fbe67c4151ded52737cb30c75429050cb7cd7777da23 2025-12-04T09:49:22.6904096Z deleted: sha256:d91e1a0a76e64abd7933fc58b8a553d76ac6cd82269c1bf701ade4e2c5ab58ff 2025-12-04T09:49:22.6904778Z deleted: sha256:f96fff40164ffa27ae535a4e5da2bbf2b18ec293366adc73519c830606f07baa 2025-12-04T09:49:22.6905346Z deleted: sha256:649089a3156b98c2cbf16b0af2b1ebdaa8644f9bf90534d9a1f7657b78b9f90e 2025-12-04T09:49:22.6905737Z deleted: sha256:aebabb7ea7535ad2aca44b23eb308b16d5de44f5b412574f94620d923436f42e 2025-12-04T09:49:22.6906105Z deleted: sha256:bcb08dc52ae55732fc33ab8c011347049a9279cc5f45a50d960c95d3a7cd5a3d 2025-12-04T09:49:22.6906474Z deleted: sha256:aa538cda5b22ea6e81fb69326637459a1356cce39b618657a075ce25d9124876 2025-12-04T09:49:22.6906856Z deleted: sha256:e28671ddaf02cbac7c1dcfdb2b08a40d3b7e2dab69c9a9e6c48ede3d8d550d66 2025-12-04T09:49:22.6907225Z deleted: sha256:ba797a4f759946d2fa7414073a5590cc12337aaa43197c35326ce456a3f84adf 2025-12-04T09:49:22.6907580Z deleted: sha256:25b28b66e2a53c3498a56aff9f543bf3a4641534c9e69741c673e9d018c652d8 2025-12-04T09:49:22.6907991Z deleted: sha256:860240198a7e7dcec697a58bcf979cfde64334033f2e97d4c9609277e051558c 2025-12-04T09:49:22.6908353Z deleted: sha256:39de54a9c331b8da952419605b26c35862b4cbb8c8a0967cc536a02b96eda874 2025-12-04T09:49:22.6908837Z deleted: sha256:6f78957316e0887707bc6534ea2aa43d1cd016130ec6a0d3148370b879696e1a 2025-12-04T09:49:22.6909201Z deleted: sha256:8656f7aa172ef2a37ec1524ac6f3ecd2b3fd9675a78a777b9a902d0f26047ea9 2025-12-04T09:49:22.6909574Z deleted: sha256:255344ff6c764e507ea2f21fbbdc6e0a1aaab0e5e0835935494c9df7d620e687 2025-12-04T09:49:22.6909944Z deleted: sha256:b6a1addb4370e1500d2fd65255b80922ad308c95a846a3460be730442d75fbe4 2025-12-04T09:49:22.6910382Z deleted: sha256:8dafa1eff3c9aada8b6725626ba0dd7de62719fb5ff307be3005cf2c6e72e2df 2025-12-04T09:49:22.6910760Z deleted: sha256:9e3fde92e8ff4f05cadac548766ef5ef9e8beb3c89cb48b4e2a17843ff60d649 2025-12-04T09:49:22.6911128Z deleted: sha256:7169a6f70d1f624733e52107b4913effe35e1e63c99388164b0afd2345658178 2025-12-04T09:49:22.6911485Z deleted: sha256:4c0d4c67766b690c1f1834310cd3898a900cd9742b7157de7e8f64d651600e35 2025-12-04T09:49:22.6911849Z deleted: sha256:7343a9c5c68976b0328278e691ddbcf26f619f989ca2a51933ac8bf637d1d69c 2025-12-04T09:49:22.6912229Z deleted: sha256:b790f29be6db3408a1a53fabdc69e76bb38cef221474a4f3aef1e233bb85d6d3 2025-12-04T09:49:22.6912593Z deleted: sha256:ed0831c232a6a410214e113695ab9288063623b523a6032a05b443fa3e9a68ae 2025-12-04T09:49:22.6912947Z deleted: sha256:554ed49d988a9f7a1381d193479542175c40d702a22d246ecef7958a8662a2eb 2025-12-04T09:49:22.6913301Z deleted: sha256:47b04d260bfb2b1f68a2f12aa70926ae2a2a53be6f4d8811d6fbe17fd7f13e17 2025-12-04T09:49:22.6913672Z deleted: sha256:6eb59e87109db04da60df057ff6b232045b6df4e2729ce7534a7a77fbe663761 2025-12-04T09:49:22.6914041Z deleted: sha256:0d9763e3d6e83eee92752c3a5dcfce74703dfa0aba42042761d7600edd95baa5 2025-12-04T09:49:22.6914386Z deleted: sha256:5811882cf3f14a50f47dc91a87d1be09daf3c20494aaf7e7343554066e9f90b7 2025-12-04T09:49:22.6914723Z deleted: sha256:4a6f4f545e9062949802a6c597609ef772633eae7d6925d41823d2b1a8991d4d 2025-12-04T09:49:22.6915069Z deleted: sha256:69a716cb9dc95d7d58d1416f4d127e49885c5d96882dc593686a6eb97200582a 2025-12-04T09:49:22.6915421Z deleted: sha256:0542610effe51f46308d4570438af559e4f6a263f38714ea04c78e4661d1909a 2025-12-04T09:49:22.6915762Z deleted: sha256:86dcdb716a38a00a400b483f42e4ac58714a70e89f5671b50ca5c45d8f577217 2025-12-04T09:49:22.6916128Z deleted: sha256:a9aaf6de8156c1b1a8ece816adbc7d2ed6edabb95e36732737ca293b26619b40 2025-12-04T09:49:22.6916492Z deleted: sha256:57ce7fcbaaa86795f8f7e6290c0aaa8622e13c9c82191d34952c0a4d2e203326 2025-12-04T09:49:22.6916852Z deleted: sha256:a90408b5f4c06a3ba881facc34fa55beb4ae4e51823e7f8d933e611f68465f4a 2025-12-04T09:49:22.6917208Z deleted: sha256:1ac1aab20bc98902f03ee9c657ff435af431f40a81ac9cebf1cd91d9041bc6b1 2025-12-04T09:49:22.6917566Z deleted: sha256:89e6c1c76b21fbe98c153a32627fa1b2337a8caa744b09b28253b67cb84012cb 2025-12-04T09:49:22.6917919Z deleted: sha256:3765d58e17aa81539a8e1eb3cb45914b48f8fb53f7ac12c1bf06c9390e274a93 2025-12-04T09:49:22.6918263Z deleted: sha256:6056a51594e8137d3425100cc233c8ab0f09dded2dc491ba083e8d7c0e1f1143 2025-12-04T09:49:22.6918635Z deleted: sha256:43a55512a68fc9eb9c622a73682dbf9f3c83f1b499287d9e12151f559edabe85 2025-12-04T09:49:22.6919057Z deleted: sha256:a5ad8e88a7e08768a61feeb801fb5a4c5041587feb2a2c77c35b285e869c25c9 2025-12-04T09:49:22.6919412Z deleted: sha256:55fd389078a5792f567a3592821e38fcfa0b0e114fa43248c4528c4d85cf6459 2025-12-04T09:49:22.6919751Z deleted: sha256:144567b58c97f1b5a75d54c3a1514a26ad5ed2dc7988e166316e324a76e76aa6 2025-12-04T09:49:22.6920100Z deleted: sha256:049fbb3123a92952eba68198938b7b29203fcb3d74dfc74e67f9929999576f19 2025-12-04T09:49:22.6920446Z deleted: sha256:22607d21e148d19a45f3f7afe927a15041b9a5663430f5580cc993c4bf9dd7dc 2025-12-04T09:49:22.6920788Z deleted: sha256:63a29311926064245f6d4a4661513586c8179b1cb460ceb91184c0d1114a742d 2025-12-04T09:49:22.6921133Z deleted: sha256:0072101bc0e6bbe2457f4a2d0de953ae643fc2aa6adc69bb59da9cbf6006c8a9 2025-12-04T09:49:22.6921500Z deleted: sha256:35c8f84104fcf39bc1be4682f46dc8b04cb1c5e4c26f66aac039cb42a3f08128 2025-12-04T09:49:22.6921857Z deleted: sha256:c6da65c085b8e72b2e240a3a7d79c4aa166c247737f5916587e51d40acff1ab8 2025-12-04T09:49:22.6922206Z deleted: sha256:b8b5ced5e8550748e26855aad0bb30864c4c435c64694097e63a9a686ef90ba0 2025-12-04T09:49:22.6922557Z deleted: sha256:9905f824759cfae96d29353305bc8a6130a7316db29dfc3efeb8ba9c7cbb5172 2025-12-04T09:49:22.6922915Z deleted: sha256:2b93a73a0fc57cdcfab2a04a30c9b313d84e5e447d076575212f198508242a00 2025-12-04T09:49:22.6923273Z deleted: sha256:baa73842b47530e5d1421010d895ed9059a567516a7e85b10943a6b5fe6fbe41 2025-12-04T09:49:22.6923625Z deleted: sha256:db79627a360d2322d58fb57c98268676011788961d4682ba034f17c9f5fdd855 2025-12-04T09:49:22.6923995Z deleted: sha256:8439f69d5eab9d42b15df43b938f56c690b18cd5e3884f4b80ddc9edf9e43822 2025-12-04T09:49:22.6924350Z deleted: sha256:e95ccdb10d43bbb6f2d6416bca8e5ad7d76a39985431aa5e40ff192ebeb52883 2025-12-04T09:49:22.6924694Z deleted: sha256:d878081c591fd25e8e494f4565e67f341cb3c459b21596ad8e8a707ff9ca63e6 2025-12-04T09:49:22.6925040Z deleted: sha256:2d594947cf73e58a20e36d52acf22f3d31b7a43291d7017c1ac71ea38c4aa9d1 2025-12-04T09:49:22.6925389Z deleted: sha256:933d315b5f20ac941653d01f2b896cc7710564c9b9c4bf9fd8e850115180c6f9 2025-12-04T09:49:22.6925740Z deleted: sha256:0edebbabed274b9e777a5933aaa21ed58569a412c97423c365eff4645e773981 2025-12-04T09:49:22.6926099Z deleted: sha256:744e1cebe181659b0a93ac857d4c16af4c2ce29f3227368eacd7ef18b69e06a7 2025-12-04T09:49:22.6926461Z deleted: sha256:71393f74ab60b183866b9f5995f85aaa65613968c611c8dc15d043befbac662b 2025-12-04T09:49:22.6926830Z deleted: sha256:741a07a29d03bdecf1fb4dfc8f1654a136064f14e5ded89a1497f476ff24623d 2025-12-04T09:49:22.6927212Z deleted: sha256:83df0b5b82e0926d6046012c3b5098dae9aa37177323a33d6eda09e392844b86 2025-12-04T09:49:22.6927591Z deleted: sha256:65b3d6509a37164c67b36965f4ca77ccb1a8e62bdd8f058e2d0886c82ba0940f 2025-12-04T09:49:22.6927982Z deleted: sha256:a38d6cb5dacc51c3e614f4ed42c05cdc53fc489f6e3dafc3b7d0eb39d3250908 2025-12-04T09:49:22.6928356Z deleted: sha256:c3de4f46ac5273a8218c6c5325247b0bd17f12037cfb49f5101a2b5085746704 2025-12-04T09:49:22.6928720Z deleted: sha256:7f3448d255710b28f79767ded93dcf11e0c1a941c7579d8752f4f0bc589aa0b5 2025-12-04T09:49:22.6929088Z deleted: sha256:fd33fe89098b4a8897870a9a316cd91868ab19f42f800d5b512e679356791e62 2025-12-04T09:49:22.6929462Z deleted: sha256:df2c12f62f5ceb2670bf7cb8aa53f268d11dbbfbf25f4b4d567673702cc3fc30 2025-12-04T09:49:22.6929838Z deleted: sha256:a3d7900a6180e4612708429f9c0edba3e999bb43fd67fd3406ae088ea7016a47 2025-12-04T09:49:22.6930202Z deleted: sha256:f592639a1c1bf8da30a1c21866ee22e76b33ae029ee236c7fd20b1ecc4716775 2025-12-04T09:49:22.6930578Z deleted: sha256:e11e4a9756b2bd4a6e62cde476dc36c6528d71a74e8e7570b271784bc494ab40 2025-12-04T09:49:22.6930949Z deleted: sha256:4c03ab513fc4cf698dd61fc37312e57abf7c4e44d4d4ddab126f38804fddec75 2025-12-04T09:49:22.6931321Z deleted: sha256:b4d3d8b5232e0bac87fa78b9c77997cbe8798faffce2aac259b820ab387ce4ad 2025-12-04T09:49:22.6931888Z deleted: sha256:dfda7f6e86c7217a3eb1fe089352248b5d84ca4f62734abce70beb4a6fc90779 2025-12-04T09:49:22.6932293Z deleted: sha256:00f41b846c66bc26a6381fe3663c611a2f539ae3b7e46e007436afd3335a7517 2025-12-04T09:49:22.6932700Z deleted: sha256:0f35c2638249a4525b29668586b079b4ee01f73bbe33732c9da7cfa7b6afb480 2025-12-04T09:49:22.6933066Z deleted: sha256:7441da072ac3269ddb67441b265fdf82904871335b5c19bb381d3622a95b7745 2025-12-04T09:49:22.6933441Z deleted: sha256:ba70851a3aedb994e8d7f8657f8f108bd90a19272843f389e6a8af48a61db720 2025-12-04T09:49:22.6933824Z deleted: sha256:1984ce6f5cabbd8c86386a8d9b5365923b92bc7ccb9aace9b4fe9570213bbbe1 2025-12-04T09:49:22.6934195Z deleted: sha256:73974f74b436f39a2fdb6461b1e3f7c3e41c73325776fa71d16b942a5b4a365b 2025-12-04T09:49:22.6934424Z 2025-12-04T09:49:22.6934663Z Total reclaimed space: 53.33GB 2025-12-04T09:49:22.7008756Z Post job cleanup. 2025-12-04T09:49:22.7034889Z Post job cleanup. 2025-12-04T09:49:22.8205274Z (node:72380) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. 2025-12-04T09:49:22.8206515Z (Use `node --trace-deprecation ...` to show where the warning was created) 2025-12-04T09:49:22.8318585Z Post job cleanup. 2025-12-04T09:49:22.8351958Z Post job cleanup. 2025-12-04T09:49:22.9257338Z [command]/usr/bin/git version 2025-12-04T09:49:22.9292578Z git version 2.50.1 2025-12-04T09:49:22.9326554Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/6c1d6ed6-c11f-4fe1-ac2f-acd86d9f048a/.gitconfig' 2025-12-04T09:49:22.9338220Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/6c1d6ed6-c11f-4fe1-ac2f-acd86d9f048a' before making global git config changes 2025-12-04T09:49:22.9339181Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T09:49:22.9344019Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:49:22.9392509Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T09:49:22.9432673Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T09:49:22.9757443Z Entering 'android/libs/fbjni' 2025-12-04T09:49:22.9817129Z Entering 'third_party/FP16' 2025-12-04T09:49:22.9865895Z Entering 'third_party/FXdiv' 2025-12-04T09:49:22.9926179Z Entering 'third_party/NNPACK' 2025-12-04T09:49:22.9981394Z Entering 'third_party/NVTX' 2025-12-04T09:49:23.0036431Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:49:23.0089231Z Entering 'third_party/XNNPACK' 2025-12-04T09:49:23.0161225Z Entering 'third_party/aiter' 2025-12-04T09:49:23.0218604Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:49:23.0288201Z Entering 'third_party/benchmark' 2025-12-04T09:49:23.0341826Z Entering 'third_party/composable_kernel' 2025-12-04T09:49:23.0404310Z Entering 'third_party/cpp-httplib' 2025-12-04T09:49:23.0455384Z Entering 'third_party/cpuinfo' 2025-12-04T09:49:23.0513780Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:49:23.0569595Z Entering 'third_party/cutlass' 2025-12-04T09:49:23.0633163Z Entering 'third_party/fbgemm' 2025-12-04T09:49:23.0692367Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:49:23.0747180Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:49:23.0807043Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:49:23.0858161Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:49:23.0923980Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:49:23.0981353Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:49:23.1035281Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:49:23.1092719Z Entering 'third_party/flash-attention' 2025-12-04T09:49:23.1146964Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:49:23.1206604Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:49:23.1271049Z Entering 'third_party/flatbuffers' 2025-12-04T09:49:23.1327380Z Entering 'third_party/fmt' 2025-12-04T09:49:23.1386700Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:49:23.1444955Z Entering 'third_party/gloo' 2025-12-04T09:49:23.1498553Z Entering 'third_party/googletest' 2025-12-04T09:49:23.1556479Z Entering 'third_party/ideep' 2025-12-04T09:49:23.1610135Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:49:23.1671170Z Entering 'third_party/ittapi' 2025-12-04T09:49:23.1724850Z Entering 'third_party/kineto' 2025-12-04T09:49:23.1778635Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:49:23.1830118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:49:23.1882464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:49:23.1932342Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:49:23.1985216Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:49:23.2036054Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:49:23.2093431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:49:23.2142568Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:49:23.2189983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:49:23.2242735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:49:23.2301025Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:49:23.2349697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:49:23.2410527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:49:23.2466040Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:49:23.2521590Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:49:23.2572807Z Entering 'third_party/kleidiai' 2025-12-04T09:49:23.2628703Z Entering 'third_party/mimalloc' 2025-12-04T09:49:23.2686538Z Entering 'third_party/nlohmann' 2025-12-04T09:49:23.2743090Z Entering 'third_party/onnx' 2025-12-04T09:49:23.2812267Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:49:23.2867262Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:49:23.2924542Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:49:23.2980014Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:49:23.3033277Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:49:23.3080443Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:49:23.3134644Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:49:23.3187185Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:49:23.3241809Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:49:23.3299793Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:49:23.3355109Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:49:23.3406300Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:49:23.3477222Z Entering 'third_party/pocketfft' 2025-12-04T09:49:23.3538978Z Entering 'third_party/protobuf' 2025-12-04T09:49:23.3596081Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:49:23.3650383Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:49:23.3706347Z Entering 'third_party/psimd' 2025-12-04T09:49:23.3760109Z Entering 'third_party/pthreadpool' 2025-12-04T09:49:23.3818371Z Entering 'third_party/pybind11' 2025-12-04T09:49:23.3882908Z Entering 'third_party/python-peachpy' 2025-12-04T09:49:23.3932689Z Entering 'third_party/sleef' 2025-12-04T09:49:23.3986666Z Entering 'third_party/tensorpipe' 2025-12-04T09:49:23.4037165Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:49:23.4093148Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:49:23.4146990Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:49:23.4196869Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:49:23.4244521Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:49:23.4326375Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T09:49:23.4350507Z http.https://github.com/.extraheader 2025-12-04T09:49:23.4359238Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-12-04T09:49:23.4392342Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T09:49:23.4705909Z Entering 'android/libs/fbjni' 2025-12-04T09:49:23.4739108Z http.https://github.com/.extraheader 2025-12-04T09:49:23.4774180Z Entering 'third_party/FP16' 2025-12-04T09:49:23.4810017Z http.https://github.com/.extraheader 2025-12-04T09:49:23.4847357Z Entering 'third_party/FXdiv' 2025-12-04T09:49:23.4884963Z http.https://github.com/.extraheader 2025-12-04T09:49:23.4923122Z Entering 'third_party/NNPACK' 2025-12-04T09:49:23.4960881Z http.https://github.com/.extraheader 2025-12-04T09:49:23.4995881Z Entering 'third_party/NVTX' 2025-12-04T09:49:23.5029355Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5065734Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:49:23.5102998Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5141581Z Entering 'third_party/XNNPACK' 2025-12-04T09:49:23.5178363Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5225701Z Entering 'third_party/aiter' 2025-12-04T09:49:23.5263185Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5293976Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:49:23.5325607Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5368864Z Entering 'third_party/benchmark' 2025-12-04T09:49:23.5407318Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5452527Z Entering 'third_party/composable_kernel' 2025-12-04T09:49:23.5490667Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5529869Z Entering 'third_party/cpp-httplib' 2025-12-04T09:49:23.5562665Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5601177Z Entering 'third_party/cpuinfo' 2025-12-04T09:49:23.5633345Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5667804Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:49:23.5702278Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5736128Z Entering 'third_party/cutlass' 2025-12-04T09:49:23.5769791Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5812004Z Entering 'third_party/fbgemm' 2025-12-04T09:49:23.5846735Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5884919Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:49:23.5921060Z http.https://github.com/.extraheader 2025-12-04T09:49:23.5958632Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:49:23.5993346Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6030773Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:49:23.6065648Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6102839Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:49:23.6137644Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6186695Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:49:23.6217321Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6246987Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:49:23.6280458Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6315606Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:49:23.6342988Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6386113Z Entering 'third_party/flash-attention' 2025-12-04T09:49:23.6420588Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6454523Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:49:23.6489559Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6529145Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:49:23.6562051Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6607893Z Entering 'third_party/flatbuffers' 2025-12-04T09:49:23.6643806Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6682181Z Entering 'third_party/fmt' 2025-12-04T09:49:23.6717453Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6755131Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:49:23.6790774Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6822016Z Entering 'third_party/gloo' 2025-12-04T09:49:23.6861532Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6897701Z Entering 'third_party/googletest' 2025-12-04T09:49:23.6928813Z http.https://github.com/.extraheader 2025-12-04T09:49:23.6965798Z Entering 'third_party/ideep' 2025-12-04T09:49:23.7001451Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7035078Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:49:23.7061869Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7104728Z Entering 'third_party/ittapi' 2025-12-04T09:49:23.7140019Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7172188Z Entering 'third_party/kineto' 2025-12-04T09:49:23.7207080Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7240606Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:49:23.7279696Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7309831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:49:23.7339176Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7371810Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:49:23.7408356Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7438849Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:49:23.7472815Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7507998Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:49:23.7541006Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7574739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:49:23.7608264Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7645346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:49:23.7682736Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7721310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:49:23.7758223Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7788304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:49:23.7823114Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7861480Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:49:23.7898793Z http.https://github.com/.extraheader 2025-12-04T09:49:23.7929548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:49:23.7965523Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8005815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:49:23.8036238Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8069213Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:49:23.8106701Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8146338Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:49:23.8178283Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8215433Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:49:23.8247649Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8287372Z Entering 'third_party/kleidiai' 2025-12-04T09:49:23.8321822Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8359842Z Entering 'third_party/mimalloc' 2025-12-04T09:49:23.8394263Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8430283Z Entering 'third_party/nlohmann' 2025-12-04T09:49:23.8463357Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8502899Z Entering 'third_party/onnx' 2025-12-04T09:49:23.8538450Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8582961Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:49:23.8615739Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8652414Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:49:23.8690554Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8727223Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:49:23.8759709Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8801133Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:49:23.8830487Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8865101Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:49:23.8902594Z http.https://github.com/.extraheader 2025-12-04T09:49:23.8937421Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:49:23.8968021Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9005119Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:49:23.9035476Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9066429Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:49:23.9104458Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9140640Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:49:23.9175273Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9206520Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:49:23.9240916Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9277717Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:49:23.9314791Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9353015Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:49:23.9389627Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9442798Z Entering 'third_party/pocketfft' 2025-12-04T09:49:23.9481453Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9517267Z Entering 'third_party/protobuf' 2025-12-04T09:49:23.9558101Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9595214Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:49:23.9624758Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9665883Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:49:23.9699575Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9734020Z Entering 'third_party/psimd' 2025-12-04T09:49:23.9771371Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9806797Z Entering 'third_party/pthreadpool' 2025-12-04T09:49:23.9840905Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9876167Z Entering 'third_party/pybind11' 2025-12-04T09:49:23.9909760Z http.https://github.com/.extraheader 2025-12-04T09:49:23.9951289Z Entering 'third_party/python-peachpy' 2025-12-04T09:49:23.9984772Z http.https://github.com/.extraheader 2025-12-04T09:49:24.0014244Z Entering 'third_party/sleef' 2025-12-04T09:49:24.0051541Z http.https://github.com/.extraheader 2025-12-04T09:49:24.0088516Z Entering 'third_party/tensorpipe' 2025-12-04T09:49:24.0125654Z http.https://github.com/.extraheader 2025-12-04T09:49:24.0162127Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:49:24.0193586Z http.https://github.com/.extraheader 2025-12-04T09:49:24.0228811Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:49:24.0258524Z http.https://github.com/.extraheader 2025-12-04T09:49:24.0291750Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:49:24.0326583Z http.https://github.com/.extraheader 2025-12-04T09:49:24.0363401Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:49:24.0398538Z http.https://github.com/.extraheader 2025-12-04T09:49:24.0435128Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:49:24.0465259Z http.https://github.com/.extraheader 2025-12-04T09:49:24.0527732Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.0569788Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T09:49:24.0892794Z Entering 'android/libs/fbjni' 2025-12-04T09:49:24.0915022Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T09:49:24.0930139Z Entering 'third_party/FP16' 2025-12-04T09:49:24.0955554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T09:49:24.0966579Z Entering 'third_party/FXdiv' 2025-12-04T09:49:24.0991283Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T09:49:24.1012046Z Entering 'third_party/NNPACK' 2025-12-04T09:49:24.1035525Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T09:49:24.1050698Z Entering 'third_party/NVTX' 2025-12-04T09:49:24.1070786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T09:49:24.1089521Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:49:24.1112186Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T09:49:24.1129778Z Entering 'third_party/XNNPACK' 2025-12-04T09:49:24.1152844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T09:49:24.1182843Z Entering 'third_party/aiter' 2025-12-04T09:49:24.1208356Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T09:49:24.1224912Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:49:24.1246596Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T09:49:24.1270336Z Entering 'third_party/benchmark' 2025-12-04T09:49:24.1295660Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:49:24.1308173Z Entering 'third_party/composable_kernel' 2025-12-04T09:49:24.1332163Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T09:49:24.1358127Z Entering 'third_party/cpp-httplib' 2025-12-04T09:49:24.1384810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T09:49:24.1400324Z Entering 'third_party/cpuinfo' 2025-12-04T09:49:24.1425969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T09:49:24.1447652Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:49:24.1470912Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T09:49:24.1489169Z Entering 'third_party/cutlass' 2025-12-04T09:49:24.1512245Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T09:49:24.1538921Z Entering 'third_party/fbgemm' 2025-12-04T09:49:24.1561448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T09:49:24.1578265Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:49:24.1600805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T09:49:24.1615644Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:49:24.1635614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T09:49:24.1660825Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:49:24.1683017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T09:49:24.1702838Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:49:24.1724366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T09:49:24.1750247Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:49:24.1772506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T09:49:24.1788050Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:49:24.1810063Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T09:49:24.1829987Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:49:24.1849814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T09:49:24.1865090Z Entering 'third_party/flash-attention' 2025-12-04T09:49:24.1891720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T09:49:24.1910632Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:49:24.1929540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T09:49:24.1950701Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:49:24.1972853Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T09:49:24.2004333Z Entering 'third_party/flatbuffers' 2025-12-04T09:49:24.2024413Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T09:49:24.2044655Z Entering 'third_party/fmt' 2025-12-04T09:49:24.2068008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:49:24.2089515Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:49:24.2112015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T09:49:24.2126737Z Entering 'third_party/gloo' 2025-12-04T09:49:24.2151129Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T09:49:24.2167181Z Entering 'third_party/googletest' 2025-12-04T09:49:24.2192630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:49:24.2212551Z Entering 'third_party/ideep' 2025-12-04T09:49:24.2232602Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T09:49:24.2248626Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:49:24.2270569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T09:49:24.2293870Z Entering 'third_party/ittapi' 2025-12-04T09:49:24.2318191Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T09:49:24.2331862Z Entering 'third_party/kineto' 2025-12-04T09:49:24.2356083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T09:49:24.2370817Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:49:24.2397868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T09:49:24.2412150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:49:24.2436762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T09:49:24.2453143Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:49:24.2472813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T09:49:24.2512641Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:49:24.2513481Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:49:24.2529780Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:49:24.2551105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T09:49:24.2565023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:49:24.2590687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T09:49:24.2609651Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:49:24.2636109Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T09:49:24.2651644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:49:24.2671144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:49:24.2689188Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:49:24.2712336Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T09:49:24.2730400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:49:24.2752147Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T09:49:24.2771263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:49:24.2792210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:49:24.2809224Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:49:24.2832084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:49:24.2848270Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:49:24.2870810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:49:24.2898213Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:49:24.2918465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T09:49:24.2954808Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:49:24.2955622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T09:49:24.2974182Z Entering 'third_party/kleidiai' 2025-12-04T09:49:24.2995665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T09:49:24.3012459Z Entering 'third_party/mimalloc' 2025-12-04T09:49:24.3037126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T09:49:24.3051383Z Entering 'third_party/nlohmann' 2025-12-04T09:49:24.3071351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T09:49:24.3091852Z Entering 'third_party/onnx' 2025-12-04T09:49:24.3114639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T09:49:24.3147422Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:49:24.3164888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:49:24.3189276Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:49:24.3211917Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T09:49:24.3230010Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:49:24.3253269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:49:24.3265514Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:49:24.3289858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:49:24.3309523Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:49:24.3327211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T09:49:24.3341783Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:49:24.3364147Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T09:49:24.3382835Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:49:24.3404902Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T09:49:24.3425667Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:49:24.3443085Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T09:49:24.3461362Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:49:24.3483073Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:49:24.3501000Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:49:24.3521670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:49:24.3544025Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:49:24.3566451Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:49:24.3588332Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:49:24.3609106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T09:49:24.3642730Z Entering 'third_party/pocketfft' 2025-12-04T09:49:24.3669049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T09:49:24.3688305Z Entering 'third_party/protobuf' 2025-12-04T09:49:24.3716194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T09:49:24.3739600Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:49:24.3761913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:49:24.3772690Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:49:24.3795341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:49:24.3819196Z Entering 'third_party/psimd' 2025-12-04T09:49:24.3843395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T09:49:24.3860761Z Entering 'third_party/pthreadpool' 2025-12-04T09:49:24.3885442Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T09:49:24.3901895Z Entering 'third_party/pybind11' 2025-12-04T09:49:24.3927997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:49:24.3946791Z Entering 'third_party/python-peachpy' 2025-12-04T09:49:24.3968374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T09:49:24.3985193Z Entering 'third_party/sleef' 2025-12-04T09:49:24.4010252Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T09:49:24.4028414Z Entering 'third_party/tensorpipe' 2025-12-04T09:49:24.4050267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T09:49:24.4068056Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:49:24.4091129Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:49:24.4109015Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:49:24.4127238Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T09:49:24.4147798Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:49:24.4169765Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T09:49:24.4185542Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:49:24.4208267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:49:24.4227710Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:49:24.4248406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T09:49:24.4287197Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4326252Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4356015Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4396460Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4419346Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4439755Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4467570Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4494542Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4522112Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4551953Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4577120Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4608540Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4637456Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4664792Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4689093Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4718503Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4751871Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4770370Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4799044Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4833418Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4861352Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4889738Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4916847Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4946174Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4967671Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.4994223Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5027702Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5046295Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5074512Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5106660Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5129819Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5156739Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5187307Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5209334Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5238204Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5266924Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5285496Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5320326Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5345518Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5372589Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5405919Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5435624Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5466008Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5496160Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5532835Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5565229Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5596092Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5616841Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5648290Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5670588Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5695320Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5719632Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5744943Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5767132Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5796749Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5827833Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5853974Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5882426Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5916238Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5949771Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.5975509Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6008255Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6034882Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6056711Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6088764Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6116561Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6153478Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6181680Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6208507Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6236962Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6273316Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6287150Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6324223Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6350407Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6381654Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6406740Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6433092Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6450532Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6477745Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6509976Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6537199Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:49:24.6660940Z A job completed hook has been configured by the self-hosted runner administrator 2025-12-04T09:49:24.6672654Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-12-04T09:49:24.6676181Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:49:24.6676435Z ##[endgroup] 2025-12-04T09:49:24.6756158Z [!ALERT!] Swap in detected! [!ALERT!] 2025-12-04T09:49:34.4566230Z [!ALERT!] Swap out detected [!ALERT!] 2025-12-04T09:49:49.1538292Z Cleaning up orphan processes